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1.  Introduction 


1.1  Background 

Robotic  interfaces  must  be  tailored  to  the  environment  in  which  they  are  used  and  thus,  it  is 
important  to  ensure  that  interfaces  for  dismounted  warfighters  are  small  and  lightweight.  A 
typical  load  for  dismounted  warfighters  can  be  well  over  100  lb  of  ammunition  and  equipment. 
Injuries  frequently  occur  just  from  carrying  this  burden  over  long  distances  in  rugged  and 
challenging  terrain  and  environments.  Dismounted  warfighters  cannot  physically  carry  the 
relatively  large  controllers  currently  used  in  stationary  environments  along  with  their  typical 
basic  loads. 

Space  to  incorporate  controls  on  small-size  controllers  is  very  limited.  Miniaturizing  individual 
controls  as  controller  sizes  get  smaller  is  not  always  an  option  as  Soldiers  wearing  gloves  have  to 
be  able  to  operate  them  individually  without  accidental  activation  of  adjacent  controls.  Thus  the 
designers  of  controllers  must  be  creative  during  functional  mapping  and  this  often  drives  them  to 
the  use  of  multifunction  controls.  The  problems  with  multifunction  controls  are  that  they  can 
increase  control  activation  time  and  increase  the  cognitive  complexity  of  the  controller.  Other 
creative  approaches  to  controller  size  reduction  that  have  been  developed  in  the  past  are  sketch 
interfaces  (i.e.,  using  hand  drawn  sketches  to  control  a  team  of  robots)  (Skubic  et  al.,  2003; 
Setalaphruk  et  al.,  2003),  voice  recognition  and  synthesis  systems  (Chen  et  al.,  2006),  and  hands¬ 
free  systems  (Veronka  and  Nestor,  2001),  but  these  novel  controls  often  present  problems  of 
their  own  and  they  are  still  being  refined. 

This  experiment  is  the  third  in  a  series  of  experiments  designed  to  investigate  current  and  future 
options  for  scaling  robotic  controllers  specifically  for  use  by  dismounted  warfighters.  In  the  first 
experiment,  different  types  of  controllers  were  evaluated  with  regard  to  impact  on  workload,  and 
usability  (Pettitt  et  al.,  2008).  The  controllers  used  were  a  multifunction  control  mounted  on  a 
weapon,  a  game  controller  with  reduced  control  sizes,  and  a  larger  robot  legacy  controller. 
Findings  indicated  that  the  multifunctional  controller  was  more  difficult  to  learn  and  use  than  the 
controller  with  the  reduced  control  sizes,  most  likely  due  to  controller  complexity  and  because 
switching  between  functions  was  time  consuming.  Also,  many  tasks  often  performed 
simultaneously  with  the  legacy  system  (i.e.,  raising  the  robot  arm  while  driving)  could  not  be 
accomplished  with  the  multifunctional  controller. 

In  the  second  experiment,  the  touch  screen  interface  on  an  Android  phone  was  compared  to  a 
game  controller  for  remotely  controlling  (teleoperating)  robots  in  an  outdoor  and  an  indoor 
course.  Total  course  completion  times,  driving  errors,  and  off  course  errors  were  significantly 
better  for  the  game  controller  than  the  Android  touch  screen.  The  Android  touch  screen  used 
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during  that  experiment  was  extremely  sensitive,  creating  many  driving  errors  and  requiring  the 
robot  to  be  backed  up  frequently  for  repositioning.  This  added  to  course  completion  times. 

Also,  the  screen  was  fairly  small  and  many  errors  were  made  because  of  inadvertent  button 
activation.  Recommendations  resulting  from  the  experiment  included  investigation  of  other 
touch  screens  (i.e.,  those  with  larger  buttons  to  accommodate  human  fingers  and  those  with 
buttons  that  are  placed  farther  apart  than  the  Android  allowed)  and  use  of  more  autonomous 
control  approaches  (Pettitt  et  al.,  2011) 

In  this  experiment  we  chose  two  controllers  that  are  commonly  used  and  have  the  potential  to 
allow  a  dismounted  Soldier  to  control  a  small  robotic  platform  and  its  sensor  payload.  Because 
of  the  difficulty  experienced  in  the  second  experiment  when  a  relatively  small  (3.7  in)  Android 
touch  screen  display  was  used  that  combined  the  display  and  control  functions  in  one  space,  we 
chose  a  larger  (12.1-in)  touch  screen  controller.  In  addition,  this  touch  screen  controller  allows 
supervisory  control  of  the  robot,  as  well  as  teleoperation.  During  supervisory  control,  the  robot 
performs  a  task  autonomously  based  on  an  input  from  the  operator,  and  the  operator  can  resume 
manual  control  as  needed.  Supervisory  control  reduces  the  number  of  fine,  sensitive 
manipulations  required  to  complete  a  mission,  while  the  larger  touchscreen  should  enable  finer 
and  more  sensitive  manipulations  when  manual  control  is  required.  Because  of  the  promise 
shown  by  the  gaming  controller  in  the  first  experiment,  we  again  chose  a  gaming  controller  as 
the  baseline  for  this  experiment.  Both  controllers  were  programmed  to  provide  the  same 
functions.  Pretest  experimentation  was  performed  to  ensure  that  the  functional  mapping  of  each 
of  the  controllers  was  as  effective  as  possible.  In  this  report  we  describe  and  discuss  tradeoffs 
between  interface  size  and  weight  with  these  controllers  vs.  input  speed,  accuracy,  training  time, 
cognitive  load,  impact  on  workload,  and  usability. 

1.2  Objective 

The  goal  of  this  research  was  to  assess  the  Soldiers’  abilities  to  effectively  perform  reconnaissance 
tasks  with  a  robot  using  a  touch  screen  interface  as  compared  to  a  commercial  off-the-shelf 
(COTS)  gaming  controller,  each  with  two  levels  of  autonomy. 

1.3  Overview  of  Experiment 

This  study  was  an  investigation  of  the  effect  of  controller  scalability  on  robotic  control.  It  took 
place  at  Fort  Benning,  GA,  and  22  Soldiers  from  the  Officer  Candidate  School  (OCS) 
participated  in  the  study.  Two  types  of  control  conditions  (i.e.,  a  virtual  joystick  [VJ]  touch 
screen  controller  and  a  handheld  game  controller  [GC])  and  two  levels  of  autonomy  (manual 
control  [MC]  with  no  click  to  navigate  and  click  to  navigate  control  [CN])  were  evaluated.  A 
2x2  within-subjects  design  was  used,  with  the  four  conditions  designated  as  GC/MC,  GC/CN, 
VJ/MC,  and  VJ/CN.  After  training  on  the  operation  of  the  robot  system,  the  Soldiers  completed 
an  outdoor  robotic  reconnaissance  course,  once  with  each  controller  type.  The  sequence  of 
controller  type  was  counterbalanced  to  control  for  the  effect  of  learning.  Controller  type  and 
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usability  were  evaluated  based  on  objective  performance  data,  data  collector  observations,  and 
Soldier  questionnaires. 


2.  Method 


2.1  Participants 

Twenty-four  Soldiers  from  the  Fort  Benning  OCS  participated  in  the  assessment.  Two  Soldiers 
served  as  participants  in  pilot  testing  and  twenty-two  for  the  conduct  of  the  experiment.  The 
OCS  participants  included  Soldiers  with  prior  enlisted  service  who  had  a  variety  of  backgrounds 
and  experience  levels,  as  well  as  those  just  coming  into  the  Army  from  college. 

2.1.1  Pretest  Orientation 

The  Soldiers  were  given  an  orientation  on  the  purpose  of  the  study  and  what  their  participation 
would  involve.  They  were  briefed  on  the  objectives,  procedures,  and  the  robotic  system.  They 
were  also  told  how  the  results  would  be  used  and  the  benefits  the  military  could  expect  from  this 
investigation.  Any  questions  the  subjects  had  regarding  the  study  were  answered. 

2.2  Apparatus  and  Instruments 

2.2.1  iRobot  310  Small  Unmanned  Ground  Vehicle*  (SUGV) 

The  iRobot  310  SUGV  (figure  1)  is  a  tactical  mobile  robot  used  to  gather  situation  awareness 
information,  often  in  areas  that  are  inaccessible  or  too  dangerous  for  humans.  The  robot  is 
equipped  with  an  Awarehead1’  payload  that  has  a  rotating  pan-and-tilt  head  equipped  with 
multiple  cameras.  The  Awarehead  features  include: 

•  Mission  tasking  control  that  enables  autonomous  control. 

•  Map  and  video  based  click-to-drive  with  obstacle  detection  and  avoidance. 

•  Map-based  navigation  using  a  global  positioning  system  (GPS). 

•  Retro-traverse  for  communications  loss.  If  the  robot  loses  communications,  it  retraces  its 
path  until  communications  are  restored. 

•  Indoor  and  outdoor  two-dimensional  mapping. 


iRobot  310  Small  Unmanned  Ground  Vehicle  is  a  registered  trademark  of  iRobot  Corp. 
'  Awarehead  is  a  registered  trademark  of  iRobot  Corp. 
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Figure  1.  iRobot  310  SUGV. 

2.2.2  Operator  Control  Unit  (OCU) 

An  Amrel  Rocky  DK8-M  ruggedized  tablet  computer  (figure  2)  was  used  as  the  OCU  to 
facilitate  the  four  control  conditions  evaluated  during  the  experiment. 


Figure  2.  Amrel  Rocky  DK8-M  tablet  computer. 

2.2.3  Robotic  Vehicle  Controllers 

2.2.3. 1  Game  Controller  (GC/MC) 

In  this  condition,  the  operator  manually  controlled  all  the  robotic  functions  using  a  COTS 
Microsoft  Xbox'  360  game  controller  (figure  3).  The  operator  manipulated  the  game  controller 
toggles  and  buttons  to  maneuver  the  robot  and  to  manipulate  the  camera,  flippers,  and  control 
arm. 


Xbox  is  a  trademark  of  Microsoft  Corporation. 
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Awarehead  Camera 

Display  Map  Display 


Figure  3.  Game  controller  (GC/MC). 

2. 2. 3. 2  Virtual  Joystick  (VJ/MC) 

When  operating  in  the  VJ/MC  condition,  all  robotic  functions  were  manually  controlled  through 
touch  screen  inputs.  The  operator  used  a  stylus  to  manipulate  the  VJ,  located  in  the  lower  left 
portion  of  the  display  (figure  4),  to  maneuver  the  robot.  The  touch  screen  stylus  was  also  used  to 
manipulate  the  camera,  flippers,  and  control  arm  using  the  control  menu  shown  in  figure  5. 
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Awarehead  Camera 

Display  Map  Display 


Virtual  Joystick 


Figure  4.  Virtual  joystick  (VJ/MC). 


Control  Menu 


Figure  5.  VJ/MC  control  menu. 

2. 2. 3. 3  Virtual  Joystick  with  Click  to  Navigate  (VJ/CN) 

In  addition  to  the  MC  options  of  the  VJ,  VJ/CN  enabled  the  operator  to  establish  waypoints  on 
the  map  or  select  a  location  on  the  video  display  for  the  robot  to  maneuver  to  by  touching  the 
screen  with  the  stylus  (figure  6).  Once  the  operator  selected  a  waypoint  or  location  on  the  video 
display,  the  robot  autonomously  maneuvered  to  the  selected  location. 


6 


Figure  6.  Virtual  joystick  with  click  to  navigate  (VJ/CN). 

2. 2. 3.4  Game  Controller  with  Click  to  Navigate  (GC/CN) 

In  addition  to  the  MC  options  of  the  GC,  GC/CN  enabled  the  operator  to  establish  waypoints  on 
the  map  or  select  a  location  on  the  video  display  for  the  robot  to  maneuver  to  by  touching  the 
screen  with  the  stylus  (figure  7).  Once  the  operator  selected  a  waypoint  or  location  on  the  video 
display,  the  robot  autonomously  maneuvered  to  the  selected  location. 


Figure  7.  Game  controller  with  click  to  navigate  (GC/CN). 

2.2.4  Robotic  Reconnaissance  Course 

The  robotic  reconnaissance  course  (figure  8)  was  -200  m  in  length,  with  five  stations  dispersed 
along  a  route.  The  stations  included  a  bunker,  tent,  foxhole,  tunnel,  and  a  zigzag  path.  Each 
station  had  a  simulated  unexploded  ordnance  (UXO)  located  in  or  around  them.  Obstacles  were 
placed  around  the  entrance  of  the  stations  requiring  the  operator  to  maneuver  the  robot  into 
position  to  locate  the  UXO.  Participants  were  located  out  of  the  line  of  sight  of  the  course  in  a 
stationary  position  inside  a  tent  near  the  course. 
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TENT 


Figure  8.  Robotic  reconnaissance  course. 

2.2.5  The  National  Aeronautics  and  Space  Administration-Task  Load  Index  (NASA-TLX) 

The  NASA-TLX  requires  the  user  to  rate  the  workload  of  a  device  on  a  number  of  different 
scales  and  to  assign  an  importance  weight  to  each  scale.  The  scores  on  the  workload  scales 
(mental,  physical,  temporal,  performance,  effort,  and  frustration)  can  be  combined  in  an  overall 
workload  score  (Hart  and  Staveland,  2008). 

2.2.6  Questionnaires 

The  questionnaires  were  designed  to  elicit  Soldiers’  opinions  about  their  performance  and 
experiences  with  each  of  the  controller  systems.  The  questionnaires  asked  the  Soldiers  to  rate 
the  devices  on  a  seven-point  semantic  differential  scale  ranging  from  “extremely  bad/difficult”  to 
“extremely  good/easy.”  Questionnaires  were  administered  to  each  Soldier  at  the  end  of  each 
iteration  (with  each  control  condition)  and  at  the  end  of  the  experiment.  Questionnaires  were 
also  used  to  gather  information  concerning  the  Soldiers’  demographic  data,  robotic  experience, 
and  physical  characteristics  that  might  affect  their  ability  to  operate  the  robot. 

2.3  Procedures 

2.3.1  Soldier  Orientation 

The  experiment  Soldiers  reported  in  groups  of  four  for  one  day  each,  from  0800  to  1700  h  daily. 
Upon  arrival,  they  received  a  roster  number  used  to  identify  them  throughout  the  evaluation. 

The  Soldiers  completed  an  informed  consent  form,  medical  status  form,  and  a  demographics 
questionnaire.  They  were  given  an  oral  operations  order  that  explained  the  robotic  mission  that 
they  would  undertake  during  the  experiment.  The  training  and  robotic  courses  were  also 
explained,  and  any  questions  the  Soldiers  had  concerning  the  experiment  were  answered. 
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2.3.2  Training 

A  representative  from  iRobot  trained  the  Soldiers  on  the  operation  of  the  iRobot  310  SUGV. 
Soldiers  practiced  teleoperating  the  robot  on  a  training  course  with  stations  similar  to  those  they 
would  encounter  on  the  reconnaissance  course  to  mitigate  learning  effects.  They  were  trained  on 
each  controller  just  before  executing  the  course  with  that  controller.  Soldiers  were  considered 
trained  once  they  were  able  to  complete  the  training  course  without  assistance.  The  average 
training  time  required  was  20  min.  A  questionnaire  concerning  the  amount  of  practice  time 
given,  the  level  of  detail  presented,  and  the  adequacy  of  training  aids  was  administered  at  the 
completion  of  training  (appendix  B). 

2.3.3  Robotic  Reconnaissance  Course  Iterations 

During  the  operations  order,  Soldiers  were  told  their  mission  was  to  conduct  a  reconnaissance  of 
five  checkpoints  plotted  on  the  OCU  map  and  report  any  UXO  found  at  each  checkpoint.  They 
were  instructed  to  reconnoiter  each  checkpoint  in  order  as  quickly  as  possible  without  hitting  any 
objects  with  the  robot.  As  a  secondary  task,  Soldiers  were  also  told  to  report  course  time  via 
radio  when  prompted. 

Soldiers  negotiated  the  course  four  times,  once  using  each  of  the  controller  types.  A  data 
collector  following  the  robot  recorded  the  time  to  complete  each  station  and  the  number  of 
driving  errors  committed.  The  operator  was  given  a  “forward”  driving  error  for  causing  the 
robot  to  hit  an  object  when  maneuvering  the  robot  forward.  If  an  object  was  hit  when  the  robot 
was  reversing,  a  “rear”  driving  error  was  recorded.  The  data  collector  with  the  robot  also  called 
the  operator  via  radio  two  times  per  station  and  asked  for  the  course  time.  A  data  collector  co¬ 
located  with  the  operator  recorded  the  amount  of  time  the  Soldier  took  to  respond  to  the 
secondary  task  and  recorded  whether  the  Soldier  stopped  the  robot  to  respond  or  was  able  to 
respond  while  simultaneously  operating  the  robot.  Table  1  shows  the  order  of  treatments  for 
each  condition.  There  were  two  levels  of  controller  (GC  vs.  VJ)  and  two  levels  of  autonomy 
(MC  vs.  CN).  After  completing  each  iteration,  the  Soldiers  were  given  a  questionnaire  designed 
to  assess  their  performance  and  experiences  with  each  of  the  control  systems.  The  participants 
also  completed  the  NASA-TLX  after  each  iteration. 

2.3.4  End  of  Experiment  Questionnaire 

After  completing  both  courses,  the  Soldiers  completed  an  end-of-experiment  questionnaire  that 
compared  each  of  the  controllers  on  a  number  of  characteristics.  They  also  completed 
questionnaires  concerning  the  information  requirements  for  teleoperating  the  robot. 
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Table  1.  Order  of  treatments. 


Roster 

Iteration 

1 

2 

3 

4 

1,22 

GC/MC 

VJ/MC 

VJ/CN 

GC/CN 

2,  23 

GC/CN 

GC/MC 

VJ/MC 

VJ/CN 

3,24 

VJ/CN 

GC/CN 

GC/MC 

VJ/MC 

4,21 

VJ/MC 

VJ/CN 

GC/CN 

GC/MC 

5,20 

GC/CN 

VJ/CN 

VJ/MC 

GC/MC 

6,  17 

GC/MC 

GC/CN 

VJ/CN 

VJ/MC 

7,  18 

VJ/MC 

GC/MC 

GC/CN 

VJ/CN 

8,  19 

VJ/CN 

VJ/MC 

GC/MC 

GC/CN 

9 

GC/MC 

VJ/CN 

VJ/MC 

GC/CN 

10 

GC/CN 

GC/MC 

VJ/CN 

VJ/MC 

11 

VJ/MC 

GC/CN 

GC/MC 

VJ/CN 

12 

VJ/CN 

VJ/MC 

GC/CN 

GC/MC 

13 

GC/CN 

VJ/MC 

VJ/CN 

GC/MC 

14 

GC/MC 

GC/CN 

VJ/MC 

VJ/CN 

15 

VJ/CN 

GC/MC 

GC/CN 

VJ/MC 

16 

VJ/MC 

VJ/CN 

GC/MC 

GC/CN 

3.  Results 


3.1  Demographics 

The  24  participants  were  recruited  from  the  OCS  located  at  Fort  Benning,  GA.  The  average  age 
of  the  Soldiers  was  29  years,  and  the  average  time  in  service  was  6  months.  None  of  the  Soldiers 
had  any  prior  military  experience  in  teleoperating  a  ground  unmanned  robot.  Detailed  responses 
to  the  demographics  questionnaire  are  available  in  appendix  A. 

3.2  Training 

Participants  stated  the  training  they  received  was  thorough  and  fully  prepared  them  to  perform 
the  tasks  required  to  conduct  the  robotic  reconnaissance  course.  Learning  to  operate  the  controls 
and  drive  the  robot  was  rated  as  being  easy  for  all  controllers.  Several  Soldiers  indicated  that  the 
hardest  task  to  learn  was  maneuvering  the  robot  in  confined  spaces  with  both  the  virtual  and 
game  control  conditions.  Several  Soldiers  commented  that  the  simplicity  of  the  Android  made  it 
easy  to  learn  to  use.  Detailed  responses  to  the  training  questionnaire  are  available  in  appendix  B. 
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3.3  Robotic  Reconnaissance  Course  Results 

During  pilot  testing,  it  was  discovered  that  the  robot’s  maximum  speed  was  less  when  the  VJ 
was  used  than  when  the  GC  was  used.  The  maximum  speed  with  the  GC  was  0.940  m/s  and 
with  the  virtual  controller  it  was  .455  m/s.  The  variations  in  speeds  were  set  within  the  software 
and  could  not  be  changed  prior  to  the  experiment.  Since  there  was  no  record  of  how  much  time 
in  each  condition  was  spent  at  maximum  speed,  course  time  data  were  not  analyzed.  Summary 
statistics  for  the  rest  of  the  performance  data  are  shown  in  table  2. 

Table  2.  Means  and  fSDs),  performance  data. 


Condition 

Errors  Forward 

Errors  Backward 

UXO  Detected 

GC/MC 

6.09  (5.39) 

1.36(1.84) 

4.95  (0.21) 

GC/CN 

7.68  (4.90) 

2.00  (2.74) 

4.91  (0.29) 

VJ/MC 

5.50  (4.26) 

1.27(1.49) 

4.68  (0.65) 

VJ/CN 

6.18(4.95) 

0.95  (1.25) 

4.82  (0.40) 

For  driving  errors  forward,  there  was  no  significant  main  effect  for  controller  (F  <  1),  no 
significant  main  effect  for  autonomy  [F(l,21)  =  2.87,  p  =  0.105,  rj'p  =  0.120],  and  no  significant 
controller  x  autonomy  interaction^  <  1).  For  driving  errors  backward,  there  was  no  significant 
main  effect  for  controller  [Ft  1,21)  =  2.48,  p  =  0.130,  //  p  =  0.106],  no  significant  main  effect  for 
autonomy  (F  <  1),  and  no  significant  controller  x  autonomy  interaction  [F(l,21)  =  2.75, 
p  =  0.1 12,  ri2p  =  0.1 16]. 

In  terms  of  number  of  UXOs  detected,  the  main  effect  for  controller  approached  statistical 
significance,  F(l,21)  =  4.05,  p  =  0.057,  >/  2P  =  0.162.  The  mean  for  the  GC  was  4.93  (Standard 
Deviation  [SD]  =  0.25),  while  the  mean  for  the  VJ  conditions  was  4.75  (SD  =  0.53).  The  main 
effect  for  autonomy  was  not  significant,  (F  <  1),  nor  was  the  controller  x  autonomy  interaction 
[F(l,21)=  1.65,/?  =  0.213,  tj2p  =  0.073], 

Table  3  summarizes  the  secondary  task  data.  For  the  response  latency  data,  there  was  no 
significant  main  effect  for  controller,  F(l,19)  =  1.70,  p  =  0.208,  //  2P  =  0.082.  There  was, 
however,  a  significant  main  effect  for  autonomy,  F(l,19)  =  7.13,  p  =  .015,  >/  2P  =  0.273. 
Latencies  were  significantly  shorter  in  the  MC  conditions  (M  =  2.9,  SD  =  0.7)  than  in  the  CN 
conditions  (M  =  3.4,  SD  =  1.0).  The  controller  x  autonomy  interaction  was  not  statistically 
significant,  F(l,19)  =  1.01,  p  =  0.328,  >/2p  =  0.050. 


Table  3.  Mean  and  (SD),  secondary  task  requests. 


Controller 

Latencies 

(s) 

Stops 

GC/MC 

2.8  (0.8) 

8.80  (2.17) 

GC/CN 

3.2  (0.8) 

4.15  (3.03) 

VJ/MC 

2.9  (0.7) 

5.75  (4.14) 

VJ/CN 

3.6  (1.1) 

3.50  (2.93) 
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For  the  number  of  stops  (figure  9)  both  the  main  effects  for  controller  [F(l,19)  =  7.36,  p  =  0.014, 
rj2p  =  0.279]  and  autonomy  [F(l,19)  =  21.3, p  <  .001,  q2p  =  0.529]  were  statistically  significant. 
In  addition,  the  controller  x  autonomy  interaction  was  statistically  significant,  F(l,19)  =  4.75, 
p  =  0.042,  rj  p  -  0.200.  Post-hoc  paired  /-tests  were  conducted  using  the  Bonferroni  correction 
for  multiple  comparisons.  With  the  CN  feature,  there  was  no  significant  difference  in  the  number 
of  stops  between  the  GC  and  the  VJ  controllers,  t  <  1 .  With  MC,  there  were  significantly  more 
stops  with  the  Xbox  controller,  t  =  4.01,  p  <  0.05. 


Figure  9.  Mean  stops,  secondary  task. 

3.4  NASA-TLX  Results 

Table  4  shows  the  mean  NASA  TLX  scale  scores  and  total  workload  scores  for  the  four 
controller  conditions. 


Table  4.  Means  and  (SDs),  NASA  TLX. 


Controller 

Mental 

Physical 

Temporal 

Performance 

GC/MC 

41.1  (28.4) 

20.5  (20.5) 

37.3  (24.9) 

28.4  (23.8) 

GC/CN 

53.6  (24.3) 

24.5  (19.7) 

42.5  (20.4) 

43.9  (24.1) 

VJ/MC 

50.9  (22.0) 

31.4  (24.6) 

43.0  (20.5) 

36.8  (21.1) 

VJ/CN 

52.3  (25.5) 

29.8  (24.3) 

44.3  (21.1) 

44.3  (18.8) 

Controller 

Effort 

Frustration 

Total 

— 

GC/MC 

35.9  (22.6) 

25.7(16.1) 

35.2  (20.3) 

— 

GC/CN 

46.4  (24.9) 

37.7  (25.2) 

47.3  (19.5) 

— 

VJ/MC 

48.4  (20.6) 

43.2  (23.5) 

47.4(18.7) 

— 

VJ/CN 

46.8  (22.1) 

44.1  (23.1) 

48.6  (17.4) 

— 
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The  total  workload  means  are  show  in  figure  10.  A  repeated  measures  analysis  of  variance 
(ANOVA)  yielded  a  significant  main  effect  for  controller  \F{\,2\)  =  11.6,/?  =  0.003,  rj 2 p  = 
0.356],  as  well  as  a  significant  main  effect  for  autonomy  [ F(  1,21)  =  4.54,  p  =  0.045, ;/  2P  = 
0.178].  In  addition,  the  controller  x  autonomy  interaction  was  statistically  significant,  F(l,21)  = 
4.47,  p  =  0.047, ;/  p  =  0.176.  With  the  CN  feature,  there  was  no  significant  difference  in  total 
workload  means  between  the  GC  and  the  joystick  controllers,  t  <  1.  With  MC,  mean  total 
workload  was  significantly  higher  with  the  joystick  than  with  the  GC,  t  =  4.33,  p  <  0.05.  This 
same  pattern  of  results  is  evident  on  the  NASA  TLX  subscales  means. 


Figure  10.  NASA  TLX  total  workload  means. 

3.5  Questionnaire  Results 

Upon  completion  of  each  iteration  of  the  robotic  reconnaissance  course,  Soldiers  were  asked  to 
rate  their  ability  to  perform  the  robotic  maneuver  tasks  using  each  controller.  The  tasks  were 
rated  using  a  seven-point  scale  with  one  being  extremely  difficult  and  seven  being  extremely 
easy.  Table  5  shows  the  maneuver  tasks  and  ratings  for  each  controller.  The  GC  received  the 
highest  rating  for  every  task  except  for  the  ability  to  multitask  (drive  and  communicate  on  the 
radio).  The  ratings  indicate  that  the  participants  preferred  MC  over  CN  for  performing  driving 
tasks. 
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Table  5.  Mean  maneuver  task  ratings. 


Driving  Tasks 

GC/MC 

GC/CN 

VJ/MC 

VJ/CN 

a.  Move  the  robot  in  the  correct  direction 

5.95 

5.23 

5.05 

4.50 

b.  Avoid  obstacles 

5.27 

4.36 

4.68 

4.18 

c.  Turn  quickly 

5.82 

4.64 

4.05 

4.18 

d.  Turn  precisely 

5.36 

4.18 

4.45 

4.14 

e.  Anticipate  whether  turn  radius  of  vehicle  will  allow  a  turn 

5.14 

4.23 

4.41 

4.14 

f.  Stop  quickly 

5.45 

5.27 

5.27 

5.05 

g.  Get  up  to  speed  quickly 

6.00 

5.82 

3.91 

4.55 

h.  Adjust  to  effects  of  system  latency  (lag) 

4.95 

4.00 

4.50 

3.73 

i.  Adjust  to  the  feel  of  the  control  system 

5.77 

5.23 

4.95 

4.36 

j.  Use  the  map  to  plan  route  in  advance 

5.55 

5.18 

4.50 

5.00 

k.  Drive  at  slowest  speeds 

4.77 

4.86 

5.45 

5.36 

1.  Drive  at  medium  speeds 

5.23 

5.09 

4.77 

4.95 

m.  Drive  at  fastest  speeds 

5.86 

5.41 

4.05 

4.45 

n.  Finish  the  course  quickly 

5.59 

5.05 

3.91 

4.32 

o.  Responsiveness  of  the  robot  to  your  commands 

5.64 

4.86 

4.68 

4.5 

p.  Drive  a  straight  route 

5.55 

5.14 

5.00 

4.68 

q.  Drive  a  route  with  multiple  waypoints 

5.41 

4.77 

4.82 

4.68 

r.  Maintain  situation  awareness 

5.59 

5.32 

4.82 

4.59 

s.  Navigate  to  the  next  waypoint 

5.59 

4.73 

4.91 

4.82 

t.  Ability  to  multitask  (drive  &communicate  on  the  radio) 

4.68 

4.45 

5.00 

4.32 

u.  Overall  ability  to  perform  this  mission 

5.77 

4.91 

4.68 

4.77 

Upon  completion  of  all  iterations,  Soldiers  were  asked  which  of  the  four  controller  conditions 
they  preferred  for  performing  various  maneuvering  tasks.  Table  6  shows  the  number  of 
preferences  for  each  controller  for  each  maneuver  task. 

Table  6.  Controller  preference. 


Driving  Task 

GC/MC 

GC/CN 

VJ/MC 

VJ/CN 

Avoiding  obstacles 

13 

3 

5 

1 

Accessing  terrain  for  navigability 

14 

5 

0 

1 

Driving  a  straight  route 

11 

5 

4 

2 

Driving  a  route  with  multiple  waypoints 

9 

9 

1 

3 

Performing  the  radio  task  while  using  the  controller 

6 

2 

11 

2 

Overall  route  reconnaissance 

11 

7 

3 

1 

With  the  exception  of  performing  radio  tasks  and  driving  a  route  with  multiple  waypoints,  the 
ratings  show  an  overall  preference  for  using  the  GC  to  perform  the  reconnaissance  course 
maneuver  tasks.  Several  Soldiers  stated  their  preference  for  the  GC  was  due  to  their  familiarity 
from  playing  computer  games.  Others  felt  that  the  robot  was  more  immediately  responsive  to  the 
GC’s  inputs  and,  thus,  it  was  much  easier  to  maneuver  between  closely  spaced  obstacles  with  it 
than  with  the  virtual  controller. 
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The  GC  also  allowed  the  operator  the  ability  to  manipulate  the  camera  and  maneuver  the  robot 
simultaneously,  whereas  the  two  tasks  were  performed  sequentially  with  the  VJ.  One  Soldier 
stated:  “Adjusting  the  flippers  and  then  moving  the  camera  with  VJ  control  was  time  consuming. 
I  could  do  both  at  the  same  time  with  the  GC.  I  had  to  frequently  switch  between  screens  in 
order  to  use  all  of  the  robot’s  functions.”  Also,  Soldiers  were  able  to  control  the  robot  with  the 
GC  without  looking  at  the  controller.  Soldiers  had  to  visually  pay  attention  to  where  the  stylus 
was  placed  on  the  VJ  relative  to  the  direction  they  wanted  the  robot  to  move  because  it  provided 
no  haptic  feedback. 

Some  Soldiers  had  more  difficulty  performing  the  secondary  radio  task  while  using  the  GC 
because  for  some  tasks,  both  of  their  hands  were  required  to  manipulate  the  controller.  They  had 
to  put  down  the  controller  in  order  to  respond  to  the  radio  query  forcing  sequential  performance 
of  the  primary  and  secondary  tasks.  Since  the  VJ  only  required  one  hand,  they  were  able  to 
accomplish  both  tasks  simultaneously. 

Even  though  overall  ratings  showed  a  preference  for  the  GC,  many  Soldiers  complained  that  it 
was  too  sensitive  and  caused  them  to  over  compensate  when  maneuvering  around  tight  spaces. 
Soldiers  stated  they  would  like  a  function  that  would  allow  them  to  adjust  the  controller 
sensitivity  similar  to  adjusting  a  mouse  curser  on  a  computer. 

Several  Soldiers  stated  that  using  the  stylus  was  tedious  and  their  hands  and  arms  became 
fatigued  during  the  VJ  iterations.  One  Soldier  also  complained  that  his  hand  that  held  the  stylus 
often  obscured  the  video  display.  Several  Soldiers  stated  that  they  would  have  preferred  to  have 
the  display  lay  flat  on  the  table  when  using  the  VJ  but,  due  to  overheating,  the  display  was 
propped  up  at  an  angle  to  allow  for  heat  dissipation.  They  had  to  visually  pay  attention  to  where 
the  stylus  was  placed  on  the  VJ  relative  to  the  direction  they  wanted  the  robot  to  move.  Many  of 
the  Soldiers  noticed  that  the  virtual  controller  did  not  allow  them  to  drive  as  fast  as  they  could 
with  the  GC  and  complained  that  this  adversely  affected  their  driving  speeds. 

The  participants  generally  preferred  MC  over  CN,  although  some  found  the  ability  to 
autonomously  maneuver  the  robot  provided  them  more  freedom  to  perform  a  secondary  task. 
After  completing  the  reconnaissance  course  with  the  VJ  with  CN,  one  Soldier  stated:  “This 
combination  represents  the  most  user  friendly  of  the  ones  tested  today.  I  could  cover  large  areas 
quickly,  multi-task,  and  make  small  adjustments  to  avoid  obstacles.”  Similar  comments  were 
made  for  both  of  the  CN  conditions.  Several  Soldiers  also  noted  that  the  CN  function  would  be 
most  useful  when  large  distances  had  to  be  traversed.  Soldiers  that  had  problems  with  the  CN 
function  often  said  that  while  it  was  great  for  long  distances  in  open  terrain,  it  was  difficult  to  use 
for  short  distances  and  when  sudden  maneuvers  were  required  because  they  were  unable  to  see 
obstacles  far  enough  in  advance  to  avoid  them.  Others  complained  that  when  the  robot  was 
operating  in  the  autonomous  mode  with  the  CN,  imprecise  or  inaccurate  GPS  data  took  the  robot 
off  course,  causing  them  to  constantly  re-click. 
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Soldiers  experienced  occasional  video  lag  in  all  four  conditions.  The  latency  between  control 
inputs  and  the  robot’s  response  sometimes  caused  driving  errors.  The  lag  was  caused  lapses  in 
communication  between  the  OCU  and  robot.  Soldiers  also  occasionally  experienced  a  shift  in 
GPS  locations  in  the  CN  conditions.  The  GPS  shifts  caused  the  robot  to  take  unintended  routes 
often  resulting  in  errors.  The  GPS  shifts  were  caused  by  poor  satellite  coverage. 

Detailed  responses  to  the  post  iteration  and  end  of  experiment  questionnaires  are  available  in 
appendices  C  and  D. 


4.  Limitations 


The  results  of  this  experiment  were  adversely  affected  by  the  inability  to  adjust  the  robot’s 
maximum  speed  to  be  the  same  with  the  virtual  controller  and  the  GC.  This  added  an 
uncontrolled  variable  to  the  course  completion  time  data  and  the  completion  times  should  have 
been  important  performance  measures.  Also,  radio  communication  between  the  OCU  and  robot 
was  adversely  affected  when  the  robot  was  not  in  line  of  sight  of  the  OCU  antenna.  Line  of  sight 
obstacles  restricted  where  waypoints  could  be  located  and  limited  the  distances  between  them. 
These  limitations  did  not  allow  a  full  evaluation  of  the  usefulness  of  the  CN  features  on 
performance  because  the  short  distances  did  not  free  up  long  enough  periods  of  operator  time  to 
show  increases  in  secondary  task  performance. 


5.  Discussion  and  Recommendations 


Although  there  was  no  significant  difference  in  terms  of  driving  errors  between  the  controller 
conditions,  Soldiers  expressed  an  overall  preference  for  the  GC  for  manual  control  of  the  robot. 
The  rubber  thumb  toggles  are  intuitive  for  driving  the  robot;  this  is  especially  true  for  the  many 
Soldiers  who  frequently  play  video  games  using  the  GC  (Pettitt,  et  al.,  201 1).  The  game  industry 
has  spent  a  lot  of  money  on  human-interface  technology  because  it  has  such  a  huge  market  and 
this  controller  has  benefited  from  significant  human  factors  research.  It  is  portable,  durable  and 
ergonomically  designed.  The  thumb  is  a  precise  pointing  instrument  and  requires  very  little 
energy  as  opposed  to  the  stylus  engaged  virtual  controller.  However,  the  simultaneous  tasks  of 
driving  the  robot  and  camera  arm  manipulation  could  not  be  performed  with  the  GC  and  its 
sensitivity  could  not  be  adjusted.  Several  Soldiers  had  difficulty  operating  the  robot  because  the 
GC  was  so  sensitive;  they  were  unable  to  make  the  robot  perform  small,  precise  movements. 

The  virtual  controller  was  less  intuitive  and  the  stylus  required  more  effort  to  use  than  the  thumb 
toggle  on  the  GC.  Use  of  the  stylus  was  necessary  because  of  the  small  size  of  the  interface. 
Another  more  Soldier- friendly  method  of  engaging  the  control  is  needed.  A  stylus  can  easily  be 
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lost,  was  uncomfortable  to  use,  and  Soldiers  also  had  to  continuously  look  at  the  virtual 
controller  when  driving  the  robot  to  ensure  that  the  stylus  was  in  the  correct  location.  Haptic 
feedback  from  the  virtual  controller  could  alleviate  the  requirement  to  look  at  the  control  since 
the  Soldier  would  be  able  to  feel  whether  the  control  was  engaged  and  which  position  of  the 
control  was  engaged  (i.e.,  north,  south,  east,  or  west). 

The  CN  technology,  which  allows  the  operator  to  select  desired  waypoint  locations,  should,  in 
theory,  require  less  involvement  from  the  operator  than  true  teleoperation  systems.  However, 
effective  CN  systems  rely  on  precise  GPS  positioning,  which  was  intermittent  during  the 
experiment. 


6.  Conclusions 


This  experiment  demonstrated  that  the  concept  of  CN  is  a  good  in  that  it  can  enhance  the 
operator’s  ability  to  multi-task.  However,  more  work  needs  to  be  done  for  it  to  perform  to  the 
level  that  is  needed  for  military  operations.  Also,  if  touch  screens  are  used  to  save  space  on 
robotic  controllers,  they  need  to  provide  haptic  feedback  so  the  operator  knows  when  a  control  is 
engaged  without  looking  at  it  or  the  robot.  They  also  need  to  be  large  enough  for  finger 
operation  since  a  stylus  is  uncomfortable  to  use  and  could  be  easily  lost  unless  it  is  tethered  to 
the  controller. 

The  GC  needs  to  allow  the  operator  to  control  the  sensitivity  of  the  controller  to  movements 
based  upon  the  operator’s  needs.  This  would  result  in  fewer  driving  errors  in  small  areas  or 
when  the  robot  is  close  to  obstacles. 
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Appendix  A.  Training 
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TRAINING 


SAMPLE  SIZE  =  22 


1 .  Using  the  scale  below,  please  rate  the  training  that  you  received  in  the  following 
areas: 


1  2  3  4  5  6  7 

Extremely  Very  Bad  Bad  Neutral  Good  Very  Good  Extremely  Good 
bad 


MEAN 

a.  Introductory  robotic  training  (comments  below) 

Completeness  of  introductory  training 

6.27 

Comprehension  of  the  overall  concept  of  the  robot 

6.36 

b.  Area  reconnaissance  training  (comments  below) 

(1)  Robotic  driving 

How  to  drive  the  robot  (Game  Controller) 

6.23 

How  to  drive  the  robot  (Virtual  Joystick) 

5.59 

How  to  drive  the  robot  (Click  to  Navigate/Virtua!  Controller) 

5.64 

How  to  drive  the  robot  (Click  to  Navigate/Game  Controller) 

5.77 

(2)  Time  provided  to  practice  driving  robot  on  route  reconnaissance 
course 

Game  Controller 

6.18 

Virtual  Joystick 

6.09 

Click  to  Navigate/Virtual  Controller) 

6.00 

Click  to  Navigate/Game  Controller) 

6.09 

(3)  Explanation  of  how  to  complete  the  route  reconnaissance  course 
(independent  of  controller  type) 

6.35 

No.  of 

Comments  Responses 


a.  Introductory  robotic  training 

Easy  to  use.  1 

Very  easy  to  pick  up  the  concepts  of  how  to  control  the  robot.  1 

There  was  plenty  of  time  for  me  to  learn  the  controls,  functions,  and 
purpose  of  the  robot.  1 

Controls  were  simple  and  training  explained  how  to  use  the  robot  very  well.  1 
Well  over  what  was  needed.  1 

Took  the  time  to  answer  my  questions.  Stopped  me  when  I  tried  to  go  too 
fast  to  explain  something.  Was  good.  1 
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b.  Area  reconnaissance  training 

Understood  completely;  easy  to  learn.  2 

I  was  told  exactly  how  to  complete  the  route  reconnaissance  course.  1 

The  instructors  were  very  informative  on  the  course,  the  robot,  and  the 
controls.  1 

I  was  given  plenty  of  explanation  of  each  feature  and  plenty  of  time  to 
practice.  2 

Adequate.  1 

Ability  to  control  is  cumbersome  (C2N/GC).  1 

Lots  of  practice.  1 

c.  Building  reconnaissance  training 

I  was  told  exactly  how  to  complete  the  building  reconnaissance  course.  1 
I  knew  it  was  being  timed,  but  I  never  felt  like  anyone  was  rushing  me  - 
very  comfortable  setting.  1 

Adequate.  1 

Sufficient  to  finish  activity.  1 

Not  much  practice  with  virtual  control.  1 

2.  What  were  the  easiest  and  hardest  training  tasks  to  learn? 

Easiest 

Fairly  easy  to  learn  how  to  use  all  of  the  interfaces  we  used  today.  1 

Easy  to  get  a  grasp  on  things  once  I  gave  it  a  try.  1 

Picking  up  speed.  With  all  controls  except  the  virtual  controller.  1 

Game  controller  concept  is  intuitive  and  most  people  know  how  to  operate 
it  already.  1 

Using  the  game  controller.  9 

Game  controller  because  all  the  controls  were  condensed  at  one  location.  1 
Game  controller  with  click  to  navigate.  1 

Click  to  navigate.  1 

Game  controller  is  exactly  same  as  game  systems  I  have  used.  2 

Virtual  controller/joystick.  1 

I  have  used  similar  click  to  navigate  systems  as  car  navigation  systems.  1 

Camera  manipulation.  1 

Robot  turning.  1 

Robot  direction,  L,  R,  forward,  back.  1 

Driving  forward  and  backwards,  turning.  1 

Steering;  controls.  1 

Hardest 

The  virtual  controller  is  probably  superior  to  the  game  controller,  but  it  took 
me  some  time  to  adjust  to  the  interface  of  the  system.  1 

Learning  to  navigate  tight  spaces  with  the  very  sensitive  game  controller.  1 
Navigation  control.  1 

Turning  in  tight  spaces  between  obstacles.  1 
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Comments 


No.  of 
Responses 


Game  controller  in  tight  spaces.  1 

Learning  how  to  navigate  the  robot  using  the  virtual  controls  in  tight 
spaces.  1 

Being  able  to  judge  how  close  I  was  to  an  object  and  if  I  had  enough  space 
around  me  to  turn.  Requiring  precise  movements  from  the  virtual  joystick 
was  hard.  1 

Click  to  navigate.  3 

Click  to  navigate:  it  seemed  that  when  I  clicked,  it  would  take  strange 
routes.  That,  combined  with  occasional  video  lag,  meant  it  was 
unpredictable  and  caused  some  driving  errors.  1 

Adjusting  to  input  lag  and  the  touch  to  navigate.  1 

Virtual.  1 

Virtual  joystick  menu.  1 

Virtual  joystick;  less  precise.  2 

Virtual  joystick  was  annoying.  1 

Virtual  controller  was  all  new  to  me.  2 

Figuring  out  virtual  controls.  1 

Putting  the  legs  down  when  extending  the  camera  forward.  1 

Virtual  control  because  the  camera  could  be  more  intuitive.  1 

Changing  position  of  the  camera  to  get  a  clear  view  of  lEDs.  1 

Controlling  camera,  head,  up,  down.  1 

Adjusting  the  head,  camera,  etc.,  without  joystick.  1 

Remember  which  buttons  did  what  on  the  game  controller.  1 

Multi-tasking  with  click  to  navigate  and  virtual  controller.  1 

Speed  with  virtual  controller.  1 

Getting  used  to  the  different  methods  of  controlling  the  robot.  1 

Situational  awareness  of  obstacles  out  of  view.  1 

General  Comments 

The  navigate/virtual  controller  offers  the  best  combination  to  the  user  to 
move  quickly,  accurately,  and  provide  the  ability  to  multi-task.  1 

Virtual  controls  were  the  smoothest  when  training.  1 

Game  controller  allowed  ease  of  access  to  all  features.  Instead  of  having 
to  switch  to  difference  screens,  if  zoom  was  also  available  on  remote 
would  be  great.  1 

Game  controller  with  click  to  navigate  is  the  best  option.  I  feel  the  game 
controller  offers  the  best  overall  control  and  speed  of  the  robot  and  the 
click  to  navigate  adds  a  short  cut  for  crossing  long  distances.  1 

Combining  the  control  types  gives  the  operator  full  control  of  the  robot.  1 
Click  to  navigate  made  controlling  the  robot  and  its  destination  point 
difficult.  1 

My  generation  was  born  with  game  controllers.  We  typically  excel  with 
video  games  and  the  familiarity  lends  itself  well  to  using  a  game  controller 
on  a  robot.  1 
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3.  What  are  your  overall  comments  on  the  training? 


No.  of 


Comments  Responses 

Training  was  very  good!  7 

I  normally  don’t  give  perfect  scores  on  surveys,  but  everyone  did  a  great 
job  of  prepping  and  explaining  for  this  experiment.  1 

Enjoyable  training.  1 

Simple,  easy  to  learn.  3 

Fun,  intuitive,  easy.  2 

Very  straightforward.  1 

Instructors  were  patient  and  helpful,  encouraging.  1 

Very  instructive  as  to  the  objectives  that  needed  to  be  accomplished.  2 

Training  allowed  me  to  manipulate  the  robot  with  relative  ease  and 
confidence.  1 

I  felt  the  training  was  thorough  and  my  understanding  of  each  controller 
was  monitored  and  corrected  during  the  testing  allowing  me  to  grow  more 
comfortable.  1 

The  different  interfaces  were  very  easy  to  learn,  much  like  teaching 
someone  how  to  use  consumer  electronics.  Soldiers  of  all  skill  levels 
should  be  able  to  learn  how  to  operate  the  system  in  a  short  amount  of 
time.  1 

The  staff  more  than  adequately  prepared  me  for  the  four  iterations,  and  I 
felt  if  I  had  to  go  down  range,  I  could  use  any  of  the  systems  if  needed.  1 
Impressive  to  see  this  technology  in  person.  1 

Robots  easy  to  control.  1 

Training  was  complete  and  I  was  provided  as  much  time  as  I  needed  to  get 
used  to  the  controls.  1 

Need  to  work  on  the  speed  sensitivity  of  the  robot  with  the  different 
controls.  1 

More  time  needed  on  virtual  control  practice.  1 

I’m  looking  forward  to  the  progress  of  IED  defeat.  1 
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Intentionally  lelt  blank. 
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Appendix  B.  Demographics 


This  appendix  appears  in  its  original  form,  without  editorial  change. 
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DEMOGRAPHICS 


SAMPLE  SIZE  =  22 


Sex 

Male  -19 
Female  -  3 


Age 

29  years 
(mean) 


MOS  Rank  Duty  Position 

09S  -  22  E-4  -  1 0  OCS  -  22 

E-5  -11 
NR  - 1 

Height  Weight 

70  inches  (mean)  171  pounds  (mean) 

(Range  is  64  to  75  in)  (Range  is  1_30  to  210  lbs) 


1 .  How  long  have  you  served  in  the  military?  6  months 

2.  How  long  have  you  had  an  infantry-related  job?  1  month 

3.  How  long  have  you  been  a  fire  team  leader?  0  months 

4.  How  long  have  you  been  a  squad  leader?  0  months 

5.  How  long  have  you  been  deployed  overseas?  1 .6  months 

6  How  long  have  you  been  deployed  in  a  combat  area?  1 .6  months 

7.  With  which  hand  do  you  most  often  write?  21  right  J_  left 

8.  With  which  hand  do  you  most  often  fire  a  weapon?  22  right  _0_  left 

9.  Do  you  wear  prescription  lenses?  1 1  No  1 1  Yes 

9.  a.  If  Yes,  which  do  you  most  often  wear?  _8_  Glasses  _2_  Contacts  J_  Both 

10.  Do  you  wear  glasses  or  contacts  while  firing  a  weapon?  JJ_Yes  1 1  No 

1 1 .  Do  you  wear  glasses  or  contacts  while  reading?  _8_  Yes  J_4_  No 

12.  Which  is  your  dominant  eye?  19  Right  3  Left 

13.  What  is  the  highest  level  of  education  you  have  achieved? 

_0_GED  _0_  High  School  _0_  Tech  School  0  Some  college 
19  College  undergraduate  degree  _3_  Master’s  degree 

14.  Where  do  you  currently  use  a  computer? 

18  Home  16  Work  _4_  Library  _2_  Other  _0_  Do  not  use 
Other:  barracks  room,  everywhere 
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15.  How  often  do  you  do/use  the  following? 


Number  of  Responses 

Daily 

Weekly 

Monthly 

Once  every 
few  months 

Rarely 

Never 

Mouse 

14 

6 

1 

0 

1 

0 

Joystick 

0 

2 

1 

3 

10 

6 

Touch  screen 

14 

2 

3 

2 

1 

0 

Software  with  icons 

19 

3 

0 

0 

0 

0 

Pull-down  menu 

17 

4 

0 

0 

1 

0 

Graphics/drawing  features 
in  software  packages* 

1 

3 

5 

4 

5 

3 

Email 

16 

4 

2 

0 

0 

0 

Operate  a  radio-controlled 
vehicle 

1 

0 

0 

2 

11 

8 

Play  first  shooter 
computer/video  games 

4 

1 

3 

4 

7 

3 

*One  soldier  had  no  response 


16.  Using  the  scale  below,  please  rate  your  skill  level  for  each  of  the  following  activities? 


12  3  4 

None  Beginner  Intermediate  Expert 


Mean  Response 

Operating  ground  unmanned  vehicles 

1.38 

Operating  aerial  unmanned  vehicles 

1.19 

Target  detection  and  identification 

1.71 

Playing  commercial  video  games 

2.62 

Training  with  Army  video  simulations 

2.00 

1 7.  How  successful  do  you  think  you  will  be  at  robotic  driving? 


Number  of  Responses 

Very  successful 

3 

Successful 

11 

Neutral 

7 

Unsuccessful 

1 

Very  unsuccessful 

0 
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Intentionally  lelt  blank. 
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Appendix  C.  Post  Iteration 


This  appendix  appears  in  its  original  form,  without  editorial  change. 
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POST  ITERATION 


SAMPLE  SIZE  =  22 

A  (GC/MC),  B  (VJ/MC),  C  (VJ/CN),  D  (GC/CN) 


1 .  Using  the  scale  below,  please  rate  your  ability  to  perform  each  of  the  following  tasks 
based  on  your  experience  with  the  autonomy  level  that  you  just  used: 

1  2  3  4  5  6  7 

Extremely  difficult  Very  difficult  Difficult  Neutral  Easy  Very  easy  Extremely  easy 


DRIVING  TASKS 

MEAN  RESPONSE 

A 

B 

C 

D 

a.  Move  the  robot  in  the  correct  direction 

5.95 

5.05 

4.50 

5.23 

b.  Avoid  obstacles 

5.27 

4.68 

4.18 

4.36 

c.  Turn  quickly 

5.82 

4.05 

4.18 

4.64 

d.  Turn  precisely 

5.36 

4.45 

4.14 

4.18 

e.  Anticipate  whether  turn  radius  of  vehicle  will  allow  a  turn 

5.14 

4.41 

4.14 

4.23 

f.  Stop  quickly 

5.45 

5.27 

5.05 

5.27 

g.  Get  up  to  speed  quickly 

6.00 

3.91 

4.55 

5.82 

h.  Adjust  to  the  effects  of  system  latency  (lag) 

4.95 

4.50 

3.73 

4.00 

i.  Adjust  to  the  feel  of  the  control  system 

5.77 

4.95 

4.36 

5.23 

j.  Use  the  map  to  plan  route  in  advance 

5.55 

4.50 

5.00 

5.18 

k.  Drive  at  slowest  speeds 

4.77 

5.45 

5.36 

4.86 

1.  Drive  at  medium  speeds 

5.23 

4.77 

4.95 

5.09 

m.  Drive  at  fastest  speeds 

5.86 

4.05 

4.45 

5.41 

n.  Finish  the  course  quickly 

5.59 

3.91 

4.32 

5.05 

o.  Responsiveness  of  the  robot  to  your  commands 

5.64 

4.68 

4.50 

4.86 

p.  Drive  a  straight  route 

5.55 

5.00 

4.68 

5.14 

q.  Drive  a  route  with  multiple  waypoints 

5.41 

4.82 

4.68 

4.77 

r.  Maintain  situation  awareness 

5.59 

4.82 

4.59 

5.32 

s.  Navigate  to  the  next  waypoint 

5.59 

4.91 

4.82 

4.73 

t.  Ability  to  multitask  (drive  and  communicate  on  the  radio) 

4.68 

5.00 

4.32 

4.45 

u.  Overall  ability  to  perform  this  mission 

5.77 

4.68 

4.77 

4.91 

30 


2.  Please  comment  on  every  task  above  that  gave  you  difficulty  and  try  to  pinpoint  the 
specific  problem  the  system  gave  you. 

No.  of 

Comments  Responses 

A 

I  liked  using  just  the  controller.  I  felt  in  control  of  the  robot  at  all  times.  The 
response  was  a  lot  better  than  the  other  method.  Getting  around  barriers 


seemed  a  lot  easier  to  maneuver.  1 

Just  put  all  controls  on  the  controller.  1 

This  mode  is  nice  for  very  close  distances  with  obstacles.  1 

All  tasks  were  not  difficult  with  the  exception  of  maneuvering  in  tight 
spaces.  I  did  not  find  this  to  be  as  fluid  of  a  motion,  more  choppy  and 
time  consuming.  1 

Compared  to  click  to  navigate,  it  does  make  multi-tasking  a  bit  more 
cumbersome  and  to  cover  large  open  areas  slower.  1 

The  overly  responsive  game  controller  was  initially  difficult,  but  I  found  that 
I  adjusted  fairly  quickly  to  using  it. 

The  control  is  hypersensitive;  overcompensated  some  times.  2 


The  game  controller  is  a  bit  too  sensitive  and  this  shows  most  in  tight 
confined  spaces,  such  as  the  tunnels  and  the  zigzag  course.  It  makes 
multiple  quick  inputs  necessary  in  order  to  correct  and  then  overcorrect 
due  to  too  much  applied  pressure.  The  function  works  very  well  as  a 


throttle,  but  certainly  affects  precise  turning.  1 

Navigating  through  obstacles  was  a  little  difficult.  It  was  easy  to  do  this  a 
low  speed,  but  more  difficult  when  you  are  trying  to  accomplish  the 
mission  quickly.  1 

Whenever  I  responded  to  radio  calls,  I  had  to  stop  moving  the  robot,  put 
down  the  controller,  and  pick  up  the  radio,  making  multi-tasking  difficult.  1 
The  most  troublesome  was  stopping  and  picking  up  the  radio.  A  headset 
would  help  a  bunch.  When  going  into  the  culvert,  it  was  hard  to  see  in.  I 
could  see  myself  running  over  something  because  of  the  lag  or  inability  to 
see.  Would  probably  be  ok  if  I  slowed  down,  but  difficult.  1 

The  turn  radius  might  be  a  little  sensitive  and  turn  too  much.  1 

It  is  tough  to  anticipate  the  turning  radius.  2 

The  X-Y  axis  of  the  camera  is  counterintuitive.  1 

Finding  the  ordinance  clearly  on  the  screen;  poor  video  and  camera 
quality.  1 

The  GPS  was  a  little  problematic;  it  didn’t  point  me  in  the  exact  direction.  1 
Robot  picks  up  speed  too  fast;  controlling  it  was  difficult.  2 

I  would  like  to  be  able  to  control  the  robot  at  a  slower  pace.  In  other 
words,  I  would  want  the  robot  to  respond  more  sensitively  to  the  controls 
when  I  want  it  to  creep  along  once  I’ve  gotten  close  to  the  objective.  1 

The  only  problem  I  experienced  was  when  the  system  lagged.  Because 
this  interface  is  a  lot  more  mobile  and  faster,  when  the  system 
experienced  lag,  it  was  hard  to  maintain  control  of  the  robot.  2 
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No.  of 
Responses 


My  issues  were  operator  errors,  not  caused  by  the  system.  1 

Hitting  the  right  turn  happens  too  quickly.  1 

It  was  difficult  to  communicate  on  the  radio  due  to  the  fact  that  both  of  your 
hands  are  on  the  controller.  Maybe  a  one-handed  controller  would  be 
best.  1 

I  experienced  a  problem  with  the  lag  time  between  the  robots  camera / 
antenna  and  the  transmitting  feed  to  the  computer.  It  made  it  harder  to 
control  the  robot’s  movements  precisely.  Every  time  the  camera  began 
to  lag,  I  stopped  moving  the  robot.  1 

On  the  course  with  the  wooden  boarder  and  the  robot  is  required  to  move 
up  to  the  edge  and  peek  over  into  a  ditch,  I  had  trouble  with  the  depth 
perception  of  how  close  the  robot’s  flippers  and  body  were  to  the  board. 

The  robot  turned  over  twice.  1 

B 

I  did  like  the  menu  system.  1 

The  time  involved  to  go  to  put  the  flippers  down  and  then  move  the  camera 
with  control;  I  could  do  both  at  the  same  time.  1 

I  would  only  use  or  want  to  use  if  I  had  many  obstacles  in  my  way  and  a 
short  distance  to  cover.  1 

Hand  gets  tired  from  the  constant  pressure  needed  on  the  screen.  It  was 
difficult  to  speed  up  the  pace  of  the  robot.  1 

Lags.  1 

Hardest  to  deal  with  the  lag.  It  caused  me  to  overcorrect  when  turning  the 
robot,  which  made  me  almost  run  into  a  few  obstacles.  1 

Holding  the  ‘pen’  hand  obstructs  view  of  navigation.  Pen  was  good  at 
slow,  precise  movements,  but  was  very  tedious  to  hold  up  to  the  screen.  1 
Navigating  many  of  the  obstacles  seemed  as  if  they  would  be  nearly 
impossible  at  a  medium  or  high  speed.  1 

Took  time  to  get  used  to  the  controls.  They  weren’t  very  precise 
sometimes;  I  would  hit  the  left  arrow  and  it  would  go  too  far  left.  Just 
difficult  to  be  accurate  with  movement.  1 

When  moving  it  to  right  or  left  and  point  off  to  the  side,  etc.,  the  robot 
appears  to  resist  in  readily  responding  to  my  wish  for  it  to  turn.  2 

Interface  is  difficult.  Small,  precise  maneuvers  required  slow  movements, 
which  seemed  to  slow  the  overall  time.  1 

Difficult  to  control.  1 

Ability  to  control  speed  and  turns  without  movement  forward.  1 

Sometimes  the  robot  adjusted  too  much  for  level  of  input.  1 

The  system  is  very  slow;  the  robot  doesn’t  move  fast.  2 

The  robot’s  responsiveness  to  changes  in  direction  were  slow.  1 

The  robot  will  not  move  faster  than  medium  speed.  1 

Hard  to  estimate  the  speed  of  robot.  1 
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No.  of 
Responses 


It  was  very  slow  to  move  between  waypoints  with  this  controller.  The 
speed  was  very  low  the  entire  time.  1 

Takes  much  longer  to  move  from  point  A  to  B,  which  increases  my 
frustration  because  I  know  this  is  a  timed  event.  1 

Traveling  from  station  to  station  with  only  the  virtual  console  is  time  and 
effort  wasted.  A  waypoint  system  is  far  more  efficient  so  the  operator 
does  need  to  focus  on  keeping  the  stylus  on  the  screen.  Other  than  that, 
it  is  accurate,  albeit  laggy  -  slow,  calculated  turns  are  simple,  but  rolling, 
fluid  ones  require  multiple  corrections  in  confined  spaces  (tunnels,  zigzag 
station).  1 

Kept  losing  GPS.  1 

Without  using  waypoint  GPS,  staying  in  waypoint  order  was  achieved  less.  1 
Using  just  the  virtual  controller,  I  had  to  frequently  switch  between  screens 
in  order  to  use  all  of  the  robot’s  functions.  In  general,  this  slowed  down 
the  speed  with  which  I  completed  the  course.  Slight  decrease  in 
situational  awareness  due  to  the  need  to  activate  a  different  menu  to  fully 
control  the  camera.  1 

Maintaining  situational  awareness  was  difficult  because  you  had  to  go 
through  several  steps  to  look  around  and  could  not  multi-task  two 
commands  such  as  moving  the  boom  arm  and  rotating  the  camera.  1 

Ability  to  identify  a  camera  focus  spot.  1 

Pointing  the  camera  in  the  right  direction  can  be  hard  when  you  have  to 
move  the  arm  where  the  main  camera  is  located.  1 

Difficult  to  adjust  the  lens/camera  to  see  targets.  1 

Difficult  to  move  the  camera,  and  time  consuming.  1 

Screen  resolution  can  be  a  hindrance.  1 

Not  handy  if  my  waypoint  is  far  and  I  needed  to  multitask.  1 

It  was  difficult  to  get  the  feel  of  this  controller  when  trying  to  avoid 
obstacles.  It  was  hard  to  turn  precisely.  1 

When  it  came  to  avoiding  obstacles,  turning  quickly  and  precise 
movements,  you  have  less  control  with  the  virtual  joystick  alone.  1 

I  prefer  the  hands-on  controller  instead;  I  feel  I  have  more  control.  1 

C 

The  virtual  controller  was  far  more  accurate  and  vastly  superior  choice  for 
any  maneuvering  around  and  amongst  obstacles.  2 

The  waypoint  function  did  work  well  and  is  the  preferred  method  of 
movement  between  locations.  Being  able  to  drag  the  line  and  create 
multiple  waypoints  makes  it  far  more  controllable  and  easier  to  use  over 
distance  than  babysitting  the  robot  with  the  virtual  controller.  1 

The  virtual  joystick  was  very  effective  for  making  precise  movements,  but 
judging  speed  was  more  difficult.  1 

I  have  to  stop  and  change  menu  screens,  which  takes  up  time  from  the 
objective.  1 
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No.  of 
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When  you  swipe  the  stylus  across  the  movement  screen,  the  robot  will 
begin  to  reset  itself  in  order  to  move.  This  takes  up  precious  time  to 
complete  each  task.  1 

When  setting  a  directional  path  on  the  map  section  of  the  screen,  the  robot 
moves  on  the  most  direct  path  possible.  The  handbrake  system  in  that 
event  is  an  excellent  idea,  however,  it’s  not  quick  enough  in  response  to 
the  robot’s  camera  to  catch  foxholes  or  pitfalls  that  are  along  the  way.  1 
Moving  in  the  correct  direction  with  the  click  to  navigate  took  several 
corrections  to  the  plots  pointed.  However,  the  joystick  was  fairly  straight 
forward.  1 

The  virtual  controller  was  a  little  difficult  to  get  used  to  and  did  not  seem  to 
be  very  precise  when  responding  to  my  commands.  1 

Adjusting  camera/lens  to  view  objects  either  below  or  above  my  line  of 
sight  gave  me  trouble.  1 

Changing  direction  of  the  camera;  not  used  to  just  pointing.  1 

Camera  could  be  more  focused.  1 

Too  much  sensitivity  at  times.  1 

I  really  don’t  like  using  the  touch  to  navigate.  The  robot  was  very  jumpy 
and  didn’t  respond  how  I  wanted  it  to.  1 

There  was  a  point  where  I  clicked  the  top  left  area  to  adjust  the  camera 
position  and  the  robot  moved  forward.  1 

I  found  that  using  this  interface  was  precise,  but  movements  are  a  lot 
slower  than  with  the  game  controller.  1 

Hard  to  control  speeds.  1 

Responsiveness  to  the  click  to  navigate  is  slow.  1 

The  robot  seemed  to  move  too  slowly  with  the  virtual  controller.  1 

Very  slow  to  get  up  to  speed  when  going  from  one  point  on  the  map  to 
another.  1 

The  speed  and  gauging  the  breaking.  1 

Difficult  to  multi-task  while  using  the  click  to  navigate.  I  set  a  point  for  the 
robot,  and  then  had  to  answer  the  radio,  but  the  robot  stayed  on  a 
straight  path  and  hit  an  obstacle.  1 

Robot  moved  to  the  waypoint  too  fast  at  times.  1 

The  straight  ahead  point  would  result  in  a  zigzag  motion.  1 

Click  to  navigate  would  not  proceed  in  straight  line  to  waypoint.  1 

The  lines  that  show  where  the  vehicle  is  going  should  reflect  the  turning 
radius  of  the  robot.  1 

Adjusting  the  neck  on  the  robot.  It’s  a  conceptual  challenge,  easier  to  do 
with  the  joystick.  1 

Switching  from  moving  the  camera  and  engaging  the  navigator.  1 

Lag  made  it  hard  in  tight  spaces.  GPS  was  off  a  little  bit,  but  click  to 
navigate  made  system  easier  to  use,  less  frustrating.  1 
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System  lag  when  initiating  click  to  navigate,  because  you  don’t  see 
obstacles  with  enough  advance  warning  so  you  can  avoid  them.  2 

The  GPS  system?  The  lag  caused  a  complete  inability  to  navigate.  1 

Click  to  navigate  isn’t  precise  enough  to  take  me  to  my  desired  point.  3 

Click  to  navigate  (by  itself)  just  isn’t  reliable  enough  for  me.  1 

It  takes  to  too  long  to  read  instructions  and  proceed.  1 

Waiting  for  the  robot  to  set  to  drive  mode.  1 

If  it  was  up  to  me,  I  would  have  driven  the  entire  course  with  the  virtual 
controller.  Seemed  the  click  to  navigate  put  me  in  the  wrong  place  and  I 
would  have  to  correct  with  the  virtual  controller  anyway.  1 

The  click  to  navigate  was  often  going  to  points  where  I  didn’t  click.  2 

Click  to  navigate  would  be  fine  if  there  were  not  so  many  obstacles.  1 

Click  to  navigate  moved  randomly  as  it  tried  to  orient  toward  the  waypoint. 

This  was  a  problem  in  tight  quarters,  with  obstacles  and  caution  tape 
nearby.  1 

Clicking  waypoints  on  the  map  itself  was  not  very  effective  because  of  the 
robot’s  tendency  to  drive  straight  through  caution  tape.  1 

Using  the  pen  when  right-handed  caused  me  to  put  the  pen  down  each 
time  to  speak  on  the  radio  because  it  wasn’t  a  good  idea  to  keep  the 
robot  in  motion  by  point-click.  1 

The  virtual  joystick  took  longer  to  adjust  instruments  and  look  around.  1 

D 

Click  to  navigate  is  great  for  long  distances,  but  difficult  to  control  for  short 
and  sudden  movements.  Great  for  speed.  1 

Using  the  game  controller  seemed  more  natural,  even  when  traveling  long 
distances  over  flat  terrain.  To  compensate  for  the  possibility  of  obstacles 
between  the  robot  and  the  waypoint,  I  would  click  on  waypoints  located 
very  close  to  the  vehicle,  resulting  in  short,  and  jerky  movements.  1 

I  challenged  myself  to  finish  quickly,  perhaps  too  quickly,  and  accordingly  I 
had  some  driving  errors  in/around  some  of  the  obstacles.  The  problem 
was  not  system  driven,  just  me  not  completely  used  to  the  system’s 
functionality  and  wanting  to  execute  quickly.  1 

A  lot  of  play  in  the  controller.  1 

Controller  sensitivity.  The  robot  responds  very  quickly  and  it’s  a  little  to 
responsive  (not  the  robot’s  fault  -  totally  a  human  problem).  1 

The  low  quality  of  the  video  imagery  made  using  click  to  navigate  difficult 
because  it  was  not  always  possible  to  see  small  obstacles  between 
waypoints  selected  and  the  robot.  1 

If  there  were  some  obstacles  near  the  robot,  it  would  stall  the  robot  in 
carrying  out  click  to  navigate;  take  too  long  in  deciding  where  to  go  or  not 
to  go  anywhere.  1 
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Hardest  part  of  this  controller  was  avoiding  obstacles  while  moving 
between  waypoints.  Since  the  robot  could  not  detect  all  obstacles,  I  had 
to  be  on  the  look-out  when  my  robot  was  moving  on  the  route  I  had  set 
for  it.  1 

Would  have  to  double  click  on  screen  to  register  destination  location.  1 

Click  to  navigate  still  had  issues.  I  loved  that  you  had  more  of  the  control 
on  the  controller.  Maybe  putting  brake  and  camera  zoom  on  the  buttons.  1 
Click  to  navigate  is  very  difficult  to  use  without  constant  input  to  adjust  the 
seemingly  random  direction  the  robot  takes  using  this  system.  It  simply 
is  not  accurate,  especially  considering  the  robot  moves  at  top  speed 
while  in  this  mode.  1 

On  click  to  navigate,  the  robot  would  spin  around,  not  going  directly  to  the 
point  you  wanted.  2 

Click  to  navigate  takes  strange  routes,  causing  you  to  use  the  emergency 
brake.  1 

The  click  to  navigate  radius  isn’t  far  enough.  1 

The  head  has  to  move  to  its  natural  position  before  click  to  move  will  move 
-  this  time  is  wasted  considering  the  game  controller  allows  movement 
regardless  of  the  head’s  position.  1 

Point  3  to  point  4  did  not  navigate  correctly  on  its  own.  Had  to  bring  the 
robot  back  on  course  manually.  1 

Moving  the  robot  using  the  click  to  navigate  was  challenging  because  the 
robot  would  not  proceed  in  a  straight  line  to  the  point.  The  response  time 
of  the  click  to  navigate  screen  was  also  a  bit  long.  2 

It  wants  to  “box”  its  way  as  it  moves  instead  of  traveling  at  an  angle  to  the 
point  on  autopilot.  1 

When  I  would  touch  the  waypoint,  the  robot  would  take  me  out  of  bounds. 

I  really  didn’t  like  using  the  point  method.  I  had  to  stop  the  robot  many 
times  because  it  was  about  to  hit  a  barrier.  I  felt  like  I  didn’t  have  very 
good  control.  Having  the  controller  as  a  back-up  helped  out;  allowed  me 
to  navigate  better.  1 

Felt  I  had  less  control  when  using  the  waypoints  because  the  robot 
seemed  to  take  unlinear  routes  and  stopping  the  robot  required  reaching 
over  and  using  the  controller.  1 

Hard  to  gauge  where  the  robot  will  eventually  end  up.  1 

Being  able  to  respond  to  radio  calls  is  difficult  when  one  hand  is  holding 
the  controller  and  another  is  holding  the  stylus.  Either  should  suffice  for  a 
soldier  to  use;  however,  the  option  to  have  both  available  is  beneficial.  1 
Click  to  navigate  sometimes  lagged  in  carrying  out  the  command;  it  would 
take  a  few  seconds  before  moving.  1 

Switching  between  controller  to  the  pen  is  a  hassle,  especially  when 
communication  was  necessary.  1 


36 


2.  Please  check  any  of  the  following  conditions  that  you  may  have  experienced  during 
this  trial. 


CONDITION 

Number  of  Responses 

A 

B 

C 

D 

Motion  sickness 

0 

0 

0 

0 

Disorientation 

0 

0 

0 

0 

Dizziness 

1 

1 

1 

1 

Other  (describe  below) 

3 

2 

2 

3 

No.  of 

Comments  Responses 

A 

Dizzy  only  because  of  time  of  day.  1 

C 

Fingers  get  tired  holding  the  stylus  from  constantly  clicking  the  screen.  1 
Everything  is  very  fuzzy;  can’t  tell  what  stuff  is.  1 

3.  Using  the  scale  below,  what  is  your  overall  rating  of  the  controller  that  you  used  this 
iteration? 

1  2  3  4  5  6  7 

Extremely  bad  Very  Bad  Bad  Neutral  Good  Very  Good  Extremely  Good 


MEAN  RESPONSE 

A 

B 

C 

D 

6.00 

4.64 

4.82 

4.86 

No.  of 

Comments  Responses 


A 

Excellent!  1 

Very  easy  to  use.  2 

Easily  adaptable  to  the  X-box  generation.  1 

Two  joysticks  are  good.  The  ability  to  operate  the  head  and  drive  at  the 
same  time  is  good.  1 

Felt  more  comfortable  with  a  controller  rather  than  a  touch  screen.  Easier 
to  use  in  my  opinion.  1 

Fast  moving  and  accurate  driving.  1 

Felt  natural  and  it  was  easy  to  control  the  vehicle.  1 

The  game  controller  is  exceptional  in  nearly  every  aspect  -  intuitive,  quick 
responding,  accurate,  and  even  ergonomic  (compared  to  poking  a  screen 
with  a  stylus).  With  all  the  buttons  mapped,  no  other  form  of  control 
would  be  necessary  (though  waypoints  are  a  nice  feature).  2 
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Great  control  and  easy  access  to  move  mechanism  on  robot.  No  thought 
is  needed  to  function/control.  1 

I  found  it  easier  to  be  more  precise  when  using  the  click  to  navigate/ 
controller  combination  to  navigate  the  course.  1 

This  controller  performed  very  well  and  responded  well  to  my  ‘commands.’ 

The  only  problem  was  that  the  robot  could  have  been  a  little  quicker  to 
respond  to  my  commands  on  the  controller;  however,  this  may  be  my 
perception  only,  and  this  may  have  been  a  result  of  ‘lag’  from  the  video 
camera.  1 

I  noticed  that  the  zoom  feature  on  the  camera  provides  more  clarity  of 
picture  to  drive  around  than  when  the  camera  is  set  to  its  normal  lens.  1 

Handy  when  you’re  close  to  target,  but  cumbersome  over  large  distances.  1 

Interface  was  a  lot  more  streamline  which  allowed  for  a  quicker  completion 
of  the  course.  It  was  a  lot  easier  to  handle  and  I  felt  like  I  exhibited  more 
control  over  the  robot.  1 

The  game  controller  is  a  bit  too  sensitive;  picks  up  speed  too  fast.  1 

Multitasking  was  difficult.  1 

From  dark  to  light,  lighting  was  a  little  difficult  to  get  over.  Growing  up  on  a 
ranch,  experience  with  heavy  equipment  helped  me  become  more  self 
aware  of  what  I  was  doing,  especially  time  spent  in  a  dozer.  1 

Had  minor  difficulty  turning  the  robot  at  minimal  speeds  precisely.  My 
suggestion  would  be  having  the  option  to  use  both  toggle  switches  to 
control  each  wheel  track  separately  for  more  control  over  steering. 

Perhaps  you  could  have  the  option  to  do  that,  and  press  the  select  button 
to  switch  back  to  have  camera  control.  1 

B 

With  only  one  hand  necessary  to  hold  the  stylus  and  control  the  robot,  it 
was  easy  to  multi-task  and  answer  the  radio.  2 

Because  of  the  low  speeds  and  ability  to  control  the  robot  with  one  hand, 
multi-tasking  with  the  radio  was  easy.  1 

Virtual  controller  made  it  easy  to  maneuver  around  close  obstacles. 

Good  control  of  the  robot  with  the  virtual  joystick.  1 

Works  well  with  minimal  lag  and  provides  accurate  control  in  confined 
areas,  as  well  as  those  with  multiple  obstacles;  just  so  long  as  you  take 
your  time  and  finesse  it  over  rushing.  1 

Compared  to  the  game  controller,  this  one  seemed  slower  for  me  to 
operate.  The  game  controller  feels  more  natural  and  I  seemed  to  have  a 
quicker  reaction  time  with  it  instead  of  this  one.  1 

I  felt  as  though  top  speed  was  slower.  1 

The  time  involved  to  move  the  camera  and  not  keep  moving  is  frustrating; 

hard  to  multi-task  controls.  1 

Not  as  good  as  the  game  controller.  It  feels  awkward  holding  the  pen  and 
placing  pressure  on  the  screen.  A  joystick  is  more  intuitive.  1 
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Responses 


I  had  to  keep  my  arm  up  to  hold  the  pen  on  the  screen  the  entire  time, 
which  made  my  arm  tired.  1 

Steering  and  turning  were  difficult.  However,  it  moved  the  robot  at  much 
lower  speeds  so  that  helped  to  maneuver  around  obstacles.  This  also 
made  it  more  difficult  to  move  between  waypoints  because  the  robot 
would  never  move  at  a  high  speed.  1 

I  would  have  given  this  a  better  rating  of  6  if  the  display  lay  flat 
(horizontally)  on  the  desk.  It  would  require  less  effort  than  working  a 
vertical  touch  screen.  1 

I  didn’t  like  using  a  stylus  for  operating  the  robot;  I  like  the  feel  of  a  joystick 
better.  1 

C 

Virtual  controller  with  waypoint  GPS  system  is  vastly  superior.  1 

Controller  allows  for  more  precise  turns  and  smaller  movements.  You  have 
100%  control  of  the  robot’s  turns  and  its  head/flippers.  1 

Using  waypoints  was  a  lot  more  efficient  than  using  the  controller  and  the 
pen.  This  system  was  very  precise  at  navigating  obstacles  but  seemed  to 
be  harder  to  control  in  the  open.  1 

Virtual  joystick  was  easier  to  be  smooth  with.  With  practice  would  be 
easier.  Plot  points  were  fast,  but  I  ran  into  some  tape.  1 

It  was  ok.  A  real  joystick  would  be  better.  1 

This  combination  represents  the  most  user  friendly  of  the  ones  tests  today. 

I  could  cover  large  areas  quickly,  multi-task,  and  make  small  adjustments 
to  avoid  obstacles.  1 

The  combination  has  a  higher  learning  curve,  and  does  not  feel  as  natural 
as  the  game  controller;  but  the  joystick  allows  for  precise  movements  and 
it  was  easier  to  multi-task  on  the  radio.  1 

I  like  the  idea  of  the  quick  navigation  using  the  point  and  click  method,  but 
the  controller  is  slow  to  respond  and  it’s  actually  easier  to  just  maneuver  it 
manually  (with  the  virtual  controller).  1 

I  felt  comfortable  using  this  controller  at  very  low  speeds  to  avoid  the 
obstacles,  but  at  medium  to  high  speeds,  I  did  not  like  the  virtual  joystick.  1 
The  touch  to  navigate  adds  speed.  1 

The  virtual  control  helps  with  accuracy  and  precision,  but  the  other  controls 
are  difficult  with  the  touch  screen.  1 

It  was  quicker  to  finish  course  by  using  click  to  navigate.  Lag  time  was 
present  but  not  enough  to  affect  results.  Easier  than  having  to  constantly 
steer  robot  forward.  1 

Click  to  navigate  is  too  worky,  not  accurate  enough.  1 

Not  reliable  or  as  accurate  as  the  controller  and  virtual  controller.  1 

Need  to  improve  camera  clarity  in  order  to  identify  objects  better.  1 

You  feel  a  loss  of  control  over  the  robot’s  actual  directional  movement 
forward  and  backward.  It’s  less  precise.  1 
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Comments 


No.  of 
Responses 


After  the  game  controller,  the  others  seem  a  little  more  challenging  to 
operate.  1 

D 

The  combination  of  click  to  navigation  and  game  controller  is  best  in  my 
opinion.  Easy  to  navigate  with  ability  to  move  automatically  without 
soldier  need  to  constantly  supervise.  Allows  for  multi-tasking  without 


losing  time.  1 

The  combination  is  a  good  one.  I  could  see  how,  if  given  enough  time  to 
practice,  I  could  get  quite  good  at  maneuvering  the  robot  quickly  and 
efficiently.  1 

The  game  controller  for  the  most  part  is  excellent,  with  few  shortcomings.  1 
The  game  controller  was  very  effective  for  precise  movements  close  to  the 
objectives.  1 

Game  controller  is  a  good  backup.  1 

Joystick  made  it  easier  to  maneuver,  better  handling,  and  control  for 
accuracy.  1 

Use  of  the  click  to  navigate  was  great  for  getting  me  on  a  straight  course.  1 
Easier  to  find  my  route  to  the  waypoint  because  I  didn’t  have  to  manually 
position  the  robot  in  the  direction  of  the  waypoint.  Once  I  set  the  point, 
the  robot  oriented  itself  to  the  point.  1 

Using  only  the  controller  was  a  lot  easier  than  switching  between  the 
controller  and  the  pen.  1 

The  click  and  move  is  very  beneficial  when  it  was  time  to  multi-task,  such 
as  with  the  radio.  1 


If  the  point  and  click  was  immediate,  it  would  be  the  best  overall  system  for 
the  plethora  of  demands  on  the  operator.  If  I  can  fully  concentrate,  I  like 
to  control  it  myself  and  of  course  I  am  faster.  If  the  lag  time  was  gone 
when  I  hit  a  waypoint  it  would  be  awesome.  If  it  could  travel  not  so  boxy 
when  it  was  on  autopilot,  it  would  even  be  better.  It  makes  me  nervous 
when  it  makes  big  “swings”  back  and  forth  when  I  am  trying  to  talk  on  the 


radio.  1 

The  lag  along  with  the  frustration  of  the  strange  routes  that  click  to  navigate 
takes  makes  the  click  to  navigate  difficult.  1 

Click  to  navigate  seems  more  appropriate  for  longer  distances.  1 

This  controller  was  more  ideal  for  the  longer  distances  between  waypoints, 
as  it  gave  me  time  to  complete  other  tasks.  However,  when  there  were 
obstacles  between  waypoints,  it  was  more  difficult  to  avoid  them  with  this 
controller.  1 

Based  on  the  grid  and  where  the  robot  eventually  stopped,  it  was  hard  to 
determine  which  square  or  point  on  the  screen  it  was  best  to  select.  1 

It  seems  easier  to  just  manually  control  the  robot.  The  good  thing  was  the 
GPS  would  get  you  going  the  correct  direction  most  of  the  time,  and  then 
it  was  better  to  control  it  manually  to  the  point.  1 


40 


Comments 


No.  of 
Responses 


The  click  to  move  is  not  ideal,  requiring  too  much  input  and  effort 
compared  to  other  methods  of  control.  1 

Using  the  touch  screen  waypoint  system  seemed  redundant.  1 

Not  reliable;  too  many  errors  (hitting  barriers,  running  out  of  bounds). 

Wasn’t  easy  to  control.  1 
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Intentionally  lelt  blank. 
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Appendix  D.  End  of  Experiment 
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END  OF  EXPERIMENT 


SAMPLE  SIZE  =  22 


1 .  Please  indicate  your  1 st,  2nd,  3rd  and  4th  choice  of  the  level  of  autonomy  to  use  for 
completing  the  following  tasks.  1st  choice  =  3  points,  2nd  choice  =  2  points,  3rd  choice  = 
1  point,  and  4th  choice  =  0  points. 


GC/MC 

VJ/MC 

VJ/CN 

GC/CN 

Avoiding  obstacles 

52 

38 

13 

29 

Driving  a  straight  route 

45 

26 

23 

37 

Driving  a  route  with  multiple  waypoints 

42 

20 

27 

43 

Performing  the  radio  task  while  using 
the  controller 

30 

47 

30 

25 

Overall  route  reconnaissance 

49 

25 

17 

41 

No.  of 

Comments  Responses 

Talking  on  the  radio  and  operating  was  the  easiest  with  virtual  controller.  1 
The  best  control  and  easiest  to  pick  up  is  the  game  controller.  1 

The  hardest  part  with  each  controller  was  avoiding  obstacles  in  tight  areas.  1 
Avoiding  obstacles  whenever  it  was  in  an  autonomous  mode.  1 

The  game  controller  with  the  ability  to  set  waypoints  is  the  ideal  set  up,  but 
adding  in  the  virtual  joystick  for  precise  turning  would  be  acceptable  in 
lieu  of  adjusting  the  sensitivity  of  the  game  controller.  1 

I  couldn’t  trust  the  click  to  navigate  to  traverse  a  route  for  waypoint  in  a 
straight  line.  1 

2.  What  suggestions  do  you  have  for  ways  to  increase  the  effectiveness  of  the: 
a.  Game  Controller? 


Comments 


No.  of 
Responses 


Game  Controller 

None,  this  is  the  best.  1 

Better  control  of  speed.  1 

Allow  a  great  variation  of  speeds.  I  wanted  to  be  able  to  move  slower  to 
navigate  obstacles.  1 

Be  able  to  control  each  wheel  track  independently  with  the  toggle  switches.  1 
Faster  responsiveness  of  robot  to  commands.  1 

Maybe  switch  to  a  one  hundred  controller.  Keep  joystick.  1 

Take  a  little  of  the  play  out.  1 
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Comments 


No.  of 
Responses 


Ability  to  plot  points,  perhaps  with  something  mounted  on  game  controller.  1 
Hands-free  communication.  1 

If  game  controller  was  only  option,  I  would  want  a  hands-free  headset  so  I 
could  communicate  with  my  soldiers.  1 

Put  all  functions  on  game  controller.  1 

Change  the  flipper  controls  so  the  top  button  is  up  and  the  bottom  button  is 
down  (that’s  how  I  accidentally  flipped  the  robot).  1 

Change  the  button  sequence  to  control  the  robot’s  neck.  R1  the  neck  could 
move  up,  and  R2,  the  neck  moves  down.  1 

Decrease  responsiveness  (sensitivity).  9 

Make  joysticks  less  sensitive.  I  found  it  difficult  to  navigate  in  tight  spaces.  1 
More  choppy  motion  than  fluid.  1 

Use  “+”  button,  not  just  joystick.  1 

A  button  on  the  game  controller  mapped  for  resuming  driving  mode  would 
be  useful.  1 

Virtual  Joystick 

Have  a  virtual  dot.  1 

Somehow  give  the  user  the  ability  to  move  much  faster  if  needed.  7 

Increasing  the  ease  of  turning  while  moving  at  speed.  The  response  time 
seemed  quite  a  bit  slow.  1 

Quicker  response  time.  1 

More  responsive  to  user’s  commands.  I  had  trouble  getting  the  robot  to 
turn  the  way  I  wanted  it  to.  1 

Mount  screen  horizontal  instead  of  vertical.  1 

I  experienced  system  lag  with  this  one  so  it’s  hard  to  say.  I  think  I’d  like  to 
try  a  horizontal  interface  as  opposed  to  the  vertical  computer  screen.  1 
Allow  direction  manipulation  on  the  screen  while  navigation  is  engaged.  1 
Add  an  auto  forward  option  -  holding  the  stylus  up  for  long  distance  is 
tedious.  1 

Add  an  additional  camera  to  monitor  the  robot  when  it’s  backing  up.  So 
when  the  joystick  is  moved  south,  the  camera  view  changes  to  reflect 
what’s  going  on  from  behind.  1 

Not  as  responsive  as  GC.  1 

Not  being  able  to  multi-task  movements.  1 

The  menu  for  controlling  the  movement  of  features  on  robot  should  be  one 
picture  of  the  robot  with  buttons  to  manipulate  each  part,  not  individual 
pictures  of  each  feature.  To  me,  it  would  be  faster  and  easier  to  use.  1 
Didn’t  like  this  one  at  all;  too  tedious  to  use.  1 

Click  to  Naviqate/Virtual  Controller 

They  both  work  together  great.  1 

Best  option.  2 

Slows  during  click  to  navigate.  Difficult  to  avoid  obstacles  when  the  robot 
is  moving  fast  on  its  own.  1 

C2N:  quicker  response  time  when  clicking  destination,  more  precise  1 
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Comments 


No.  of 
Responses 


navigation  around  obstacles. 

Fix  the  C2N  -  it  zigzags  straight  lines  and  requires  constant  babysitting  to 
guarantee  its  going  where  you  want.  1 

The  C2N  allows  speed.  1 

Make  the  C2N  not  so  jumpy;  more  accurate.  1 

Smoothing  out  the  C2N,  I  ran  into  the  tape  a  couple  of  times,  which  then 
required  me  to  direct  attention  to  over  watch  so  it  didn’t  get  into  trouble.  1 
The  C2N  was  redundant  and  unnecessary.  1 

VC:  quicker  response  time  when  clicking  destination,  more  precise 
navigation  around  obstacles.  1 

VC  allows  handling,  but  limits  the  ease  of  adjusting  the  robot,  camera, 
flippers,  etc.  1 

Allow  user  to  adjust  the  speed  of  the  robot  to  avoid  obstacles  better.  1 

More  precise  speed  and  turning  controls.  1 

I  experienced  system  lag,  as  well  but  I  think  the  combination  works  well 
(especially  in  tricky  situations  like  multiple  obstacles  and  tight  spaces).  1 
To  increase  performance,  the  controls  would  have  to  respond  more  readily 
with  less  lag  time  and  more  accuracy.  1 

Difficult  to  navigate  in  tight  terrain.  1 

Mount  screen  horizontal.  1 

Make  robot’s  movement  more  predictable.  2 

Better  system  of  determining  where  the  robot  will  end  up.  1 

With  navigation  system,  it  would  be  helpful  to  be  able  to  detect  obstacles  in 
between  large  open  areas.  1 

The  robot  seems  to  point  in  the  wrong  direction  when  using  click  to 
navigate.  1 

C2N  robot  needs  to  respond  to  your  command  in  a  straight  line.  1 

A  better  way  to  switch  between  camera  mode  and  navigate  mode.  1 

Not  being  able  to  multi-task  movements.  1 

Navigate  was  not  reliable.  1 

Click  to  Naviqate/Game  Controller 

Game  controller  is  the  best.  1 

Good  combination,  but  hard  to  maneuver  due  to  sensitivity  of  control.  1 

Waypoints  are  a  useful  addition  to  the  game  controller.  1 

Is  great  as  is  unless  you  change  the  type  of  controller  to  make  the  multi¬ 
tasking  part  less  cumbersome.  For  instance,  use  the  street  fighter  arcade 
controller  layout  (a  joystick  with  six  buttons).  1 

They  both  work  together  well,  but  it’s  tough  since  they’re  separate 
machines.  Hands  are  fumbling,  plus  the  radio  is  more  trouble.  1 

Somehow  find  an  easier  transition  between  the  pen  for  the  click  to  navigate 
and  the  joystick.  1 

Slows  during  click  to  navigate.  Difficult  to  avoid  obstacles  when  the  robot 
is  moving  fast  on  its  own.  1 
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Comments 


No.  of 
Responses 


C2N:  quicker  response  time  when  clicking  destination,  more  precise 
navigation  around  obstacles.  1 

C2N  was  a  pain  with  the  game  controller  because  I  had  to  constantly 
switch  between  the  controller  and  pen.  1 

One  hundred  controller  so  other  hand  can  be  free  for  radio  comms.  1 

More  predictable  robot  movement  during  click  to  navigate.  1 

Better  system  of  determining  where  the  robot  will  end  up.  1 

With  navigation  system,  it  would  be  helpful  to  be  able  to  detect  obstacles  in 
between  large  open  areas.  1 

Mount  screen  horizontal.  1 

I  did  not  like  this  combination  at  all.  The  bulky  controller  was  difficult  to 
handle  when  switching  back  and  forth  from  the  click  to  navigate  screen. 

A  smaller,  easier  to  handle  controller  could  help  with  that.  1 

Liked  ability  to  stop  navigation  by  simply  directing  robot  back  on  joystick, 
easy  to  stop.  1 

Allow  direction  manipulation  on  the  game  controller  while  navigation  is 
engaged.  1 

General  Comments 

Both  of  the  click  to  navigate  options  would  be  more  effective  with  higher 
quality  imagery  so  the  operator  has  a  better  idea  of  what  he  is  moving  the 
vehicle  toward.  1 
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3.  Using  the  scale  below,  what  is  your  rating  of  the  human  factors  characteristics  of 
system  that  you  used  curing  this  experiment? 

1  2  3  4  5  6  7 

Extremely  bad  Very  Bad  Bad  Neutral  Good  Very  Good  Extremely  Good 


CHARACTERISTICS 

MEAN 

a.  Display  size  for  driving 

5.14 

b.  Display  size  for  seeing  lEDs 

4.91 

c.  Size  of  the  buttons  on  the  screen 

5.27 

d.  Placement  of  the  buttons  and  controls  on  the  screen 

5.09 

e.  Sensitivity  of  the  touch  screen 

4.91 

f.  Latency/lag  in  the  system 

4.27 

g.  Ease  of  using  the  arm  controller  while  using  the  game  controller 

5.95 

h.  Ease  of  using  the  arm  controller  while  using  the  virtual  controller 

4.26 

i.  Ease  of  using  the  flippers  while  using  the  game  controller 

6.09 

j.  Ease  of  using  the  flippers  while  using  the  virtual  controller 

4.41 

k.  Usefulness  of  the  video  touch 

5.36 

1.  Display  layout 

5.18 

m.  Map  fidelity 

4.64 

n.  Overall  controller  size  for  use  in  the  field 

5.73 

o.  Controller  configuration  for  use  in  the  field 

5.73 

No.  of 


Comments  Responses 

Mostly  everything  works  great.  2 

The  best  of  all  worlds  was  the  game  controller.  1 

Display  size  was  nice.  1 

Regardless  of  system  used,  the  more  practice  given  the  better  the 
performance.  1 

Perhaps  using  the  GC  for  area  recon  and  switching  to  virtual  for  closer/tight 
space  objectives.  1 

Controls  for  the  robot  using  the  virtual  controller  were  difficult  to  access 
quickly.  1 

Video  resolution  is  poor.  1 

Putting  the  screen  down  more  like  a  tablet  may  be  more  comfortable.  1 

Overall  system  is  very  sensitive.  1 

Would  opt  for  click  to  navigate/game  controller.  Would  add  additional 
features  to  remote.  Minimize  usage  of  screen  for  additional  features 
currently  not  available  on  remote.  1 

Imagery  quality  from  the  cameras  mounted  on  the  robot  could  be  better.  1 
On  the  screen,  increase  the  size  of  the  buttons.  1 

On  the  game  controller,  a  quick  tap  to  move  the  robot  on  its  current 
heading  to  free  up  a  hand  for  another  task.  1 
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Comments 


No.  of 
Responses 


The  map  seemed  to  be  slightly  off  from  reality.  1 

While  using  joystick,  you  had  to  stop  all  other  tasks  in  order  to  manipulate 
flippers  or  arm/camera.  The  GC  allowed  for  better  multi-tasking.  1 

4.  What  needs  to  be  added  to  the  controller  to  give  you  more  knowledge  of  the  area 
around  the  robot? 


Provided  for  the  most  situational  awareness  thanks  to  the  ease  of  use  of 
the  camera.  1 

I  was  able  to  do  everything  I  needed  to  during  the  mission.  1 

Add  zoom  options  to  the  controller.  3 

Need  a  way  to  zoom  in  and  out  with  just  the  game  controller.  1 

Something  that  could  detect  obstacles  further  out  would  be  helpful.  1 

Optional  360  view  (wide  angle  front,  rear,  and  sides  of  at  same  time)  for 
maneuvering  around  obstacles.  2 

More  multiple  cameras/video  displays  open  simultaneously.  1 

Cameras  pointing  behind  to  the  sides  of  the  robot.  2 

For  quickness,  more  cameras  on  the  side  that  you  can  quickly  toggle  to. 

Hardly  used  chassis  camera  but  when  I  did,  it  was  helpful  to  orient  main 
camera.  1 

Better  cameras  with  clearer  definition.  1 

A  button  that  could  allow  fast  access  to  a  rear  camera.  1 

A  button  on  the  controller  that  allows  for  a  quick  change  in  camera  angles.  1 
A  toggle  button  to  quick  view  the  side  cameras.  Rather  than  having  to 
rotate  robot  head  to  look  to  sides.  This  would  allow  it  to  be  more  time 
efficient  and  would  allow  for  quick  security  checks.  1 

I  had  difficulty  moving  the  video  arm  of  the  robot,  especially  with  the  virtual 
controller.  1 

The  driving  display  could  have  been  better  if  it  were  larger.  1 

Height  sensors/info.  1 

Manual  focus,  sensors.  1 

A  button  on  the  controller  that  quickly  resets  the  robot  to  roving  mode.  1 


5.  Where  would  you  like  to  see  a  robotic  controller  placed? 


Number  of  Responses 

Wrist 

Chest 

Cargo  Pocket 

Other 

5 

4 

12 

4 

Maybe  a  high  tech  visor.  1 

Thigh  area  front.  1 
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Comments 


No.  of 
Responses 


Anything  that  can  be  accessed  relatively  easily  and  does  not  add  too  much 
weight.  1 

Bring  in  the  system  when  needed;  carry  the  controller  in  your  rucksack.  1 
If  you  were  running  and  moving,  it  would  be  nice  to  have  it  strapped  to  you 
so  you  would  not  lose  it  when  %$#  hits  the  fan.  Put  it  on  a  strap  that  will 
zip  it  back  up  when  finished,  so  you  can  just  drop  it  and  grab  your 
weapon  if  need  be.  The  controller  zips  up  to  a  secure  spot  on  your  chest, 


and  you  can  grab  it  when  you  next  have  a  change  to  continue  mission.  1 
6.  Overall  comments: 

Hooah!  1 

Great  experience!  3 

This  was  an  extremely  neat  experience  and  I’m  extremely  fortunate  and 
appreciative  to  have  helped.  I  enjoyed  it.  Thank  you!  1 

Controller  is  my  number  1 !  Virtual  controller  is  number  2,  and  touch  to 
navigate  is  number  3.  The  controller  is  easier  to  use  and  to  get  used  to 
quicker.  1 

System  is  good  overall,  comfortable  and  easy  to  master.  1 

Much  easier  to  control  these  robots  than  I  had  anticipated.  Just  like 
playing  a  video  game.  1 

The  robot  was  very  easy  to  use  on  the  GC.  My  suggestions  would  be  to 


allow  customization  of  the  controller  layout  to  allow  personalization  to  an 
individual  soldier;  and  also  be  able  to  control  each  wheel  track 
independently  through  both  toggles,  but  be  able  to  switch  to  camera  and 
arm  control.  The  menu  for  controlling  features  on  the  joystick  could  be 
condensed  into  a  single  picture  with  options  for  each  different  part.  This 
could  speed  identification  and  selection  of  desired  movements. 
Personally,  I  would  use  the  C2N  in  wide  open  spaces  across  long 
distances,  but  no  other  time  because  I  could  not  trust  the  system  to 
proceed  to  the  waypoint  in  a  straight  line.  The  zoom  feature  was  great  in 
the  reconnaissance  of  routes  and  buildings,  but  the  definition  and  clarity 


of  the  normal  camera  zoom  was  a  little  rough;  could  be  sharper.  1 

Overall,  the  click  to  navigate  with  game  controller  is  my  best  option.  1 

The  game  controller  was  the  best.  2 

Game  controller  has  the  most  control.  1 

Game  controller  was  easiest  to  pick  up.  1 

I  found  the  game  controller  to  be  the  most  streamline  control  system. 

However,  the  sensitivity  of  the  controller’s  joysticks  made  navigating 
obstacles  harder.  Overall,  I  preferred  the  game  controller.  1 

The  easiest  to  multi-task  was  the  virtual  land  plotting  waypoints.  1 

Did  not  care  for  the  click  to  navigate.  It  seemed  to  cause  more  issues.  It 
was  faster  to  just  drive  the  robot.  1 
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Comments 


No.  of 
Responses 


Maybe  switch  to  a  one  hundred  version  if  the  user  needs  to  have  a  hand 
free.  Add  zoom  controls  to  the  game  controller.  That  was  the  only  time  I 
had  to  take  my  hands  off  the  controller  during  iteration  3.  1 

Using  the  game  controller  or  virtual  controller  in  combination  with  click  to 
navigate  seemed  more  trouble  than  it  was  worth,  especially  at  the  close 
ranges  we  were  dealing  with  on  the  course.  The  combination  options 
would  be  more  useful  in  situations  where  the  robot  has  a  long  distance  to 
travel.  1 

A  combination  of  the  game  controller/virtual  with  plotting  points  would  be 
the  best  configuration  to  tackle  most  missions.  A  few  kinks  worked  out  of 
the  plotting  waypoints  so  it’s  smoother  in  the  auto  pilot  would  be  my  only 
suggested  improvement  for  the  plotting.  1 

Put  a  “continue  on”  trajectory  button  on  game  controller  to  facilitate  multi¬ 
tasking.  1 

If  I  was  down  range,  I  would  want  to  use  the  click  to  navigate/virtual 
controller  system.  1 


51 


Intentionally  lelt  blank. 
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List  of  Symbols,  Abbreviations,  and  Acronyms 


ANOVA 

analysis  of  variance 

CN 

navigate  control 

COTS 

commercial-off-the-  shelf 

GC 

game  controller 

GPS 

global  positioning  system 

HMI 

Human  Machine  Interface 

MC 

manual  control 

NASA-TLX 

National  Aeronautics  and  Space  Administration-Task  Load  Index 

OCS 

Officer  Candidate  School 

OCU 

operator  control  unit 

SD 

standard  deviation 

SOURCE 

Safe  Operations  of  Unmanned  Systems  for  Reconnaissance  in  Complex 
Environments 

SUGV 

Small  Unmanned  Ground  Vehicle 

UXO 

unexploded  ordnance 

VJ 

virtual  joystick 
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NO.  OF 

COPIES  ORGANIZATION 


1  DEFENSE  TECHNICAL 
(PDF  INFORMATION  CTR 
only)  DTIC  OCA 

8725  JOHN  J  KINGMAN  RD 
STE  0944 

FORT  BELVOIR  VA  22060-6218 

1  DIRECTOR 

US  ARMY  RESEARCH  LAB 
IMNE  ALC  HRR 
2800  POWDER  MILL  RD 
ADELPHI  MD  20783-1 197 

1  DIRECTOR 

US  ARMY  RESEARCH  LAB 
RDRL  CIO  LL 
2800  POWDER  MILL  RD 
ADELPHI  MD  20783-1 197 

1  DIRECTOR 

US  ARMY  RESEARCH  LAB 
RDRL  CIO  LT 
2800  POWDER  MILL  RD 
ADELPHI  MD  20783-1 197 

1  DIRECTOR 

US  ARMY  RESEARCH  LAB 
RDRL  D 

2800  POWDER  MILL  RD 
ADELPHI  MD  20783-1 197 
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NO.  OF 

COPIES  ORGANIZATION 


NO.  OF 

COPIES  ORGANIZATION 


2  ARMY  RSCH  LABORATORY  -  HRED 

HUMAN  RSRCH  AND  ENGRNG 
DIRCTRT  MCOE  FIELD  ELEMENT 
RDRL  HRM  DW 
R  PETTITT 
C  CARSTENS 
BLDG  4  RM  332 

FORT  BENNING  GA  31905-5400 

1  ARMY  RSCH  LABORATORY  -  HRED 
RDRL  HRM  C  A  DAVISON 

320  MANSCEN  LOOP  STE  1 15 
FORT  LEONARD  WOOD  MO  65473 

2  ARMY  RSCH  LABORATORY  -  HRED 
RDRL  HRM  DI 

T  DAVIS 

J  HANSBERGER 

BLDG  5400  RM  C242 

REDSTONE  ARSENAL  AL  35898-7290 

1  ARMY  RSCH  LABORATORY  -  HRED 

RDRL  HRS  EA  DR  V  J  RICE 
2377  GREELEY  RD  STE  R 
FORT  SAM  HOUSTON  TX  78234-5002 

1  ARMY  RSCH  LABORATORY  -  HRED 

RDRL  HRM  DG  K  GUNN 
BLDG  333 

PICATINNY  ARSENAL  NJ  07806-5000 

1  ARMY  RSCH  LABORATORY  -  HRED 

ARMC  FIELD  ELEMENT 
RDRL  HRM  CH  C  BURNS 
BLDG  1467B  RM  336  THIRD  AVE 
FORT  KNOX  KY  40121 

1  ARMY  RSCH  LABORATORY  -  HRED 

AWC  FIELD  ELEMENT 
RDRL  HRM  DI  D  DURBIN 
BLDG  4506  (DCD)  RM  107 
FORT  RUCKER  AL  36362-5000 

1  ARMY  RSCH  LABORATORY  -  HRED 

RDRL  HRM  CK  J  REINHART 
10125  KINGMAN  RD  BLDG  317 
FORT  BELVOIR  VA  22060-5828 


1  ARMY  RSCH  LABORATORY  -  HRED 

RDRL  HRM  AY  M  BARNES 
2520  HEALY  AVE 
STE  1172  BLDG  51005 
FORT  HUACHUCA  AZ  85613-7069 

1  ARMY  RSCH  LABORATORY  -  HRED 

RDRL  HRM  AP  D  UNGV ARSKY 
POPE  HALL  BLDG  4709 
BCBL  806  HARRISON  DR 
FORT  LEAVENWORTH  KS  66027-2302 

1  ARMY  RSCH  LABORATORY  -  HRED 

RDRL  HRM  DQ  MR  FLETCHER 
AMSRD  NSC  WS  E  BLDG  3  RM  341 
NATICK  MA  01760-5020 

1  ARMY  RSCH  LABORATORY  -  HRED 

RDRL  HRM  AT  J  CHEN 
12350  RESEARCH  PKWY 
ORLANDO  FL  32826-3276 

1  ARMY  RSCH  LABORATORY  -  HRED 

RDRL  HRM  AT  C  KORTENHAUS 
12350  RESEARCH  PKWY 
ORLANDO  FL  32826 

1  ARMY  RSCH  LABORATORY  -  HRED 

RDRL  HRM  AS  C  MANASCO 
SIGNAL  TOWERS 
BLDG  29808A  RM  303 
FORT  GORDON  GA  30905-5233 

1  ARMY  RSCH  LABORATORY  -  HRED 

RDRL  HRM  CU 
6501  E  11  MILE  RD  MS  284 
BLDG200A  1ST  FL  RMA1117 
WARREN  MI  48397-5000 

1  ARMY  RSCH  LABORATORY  -  HRED 

FIRES  CTR  OF  EXCELLENCE 
FIELD  ELEMENT 
RDRL  HRM  AF  C  HERNANDEZ 
3040  NW  AUSTIN  RD  RM  221 
FORT  SILL  OK  73503-9043 

1  ARMY  RSCH  LABORATORY  -  HRED 

RDRL  HRM  AV  S  MIDDLEBROOKS 
91012  STATION  AVE 
FORT  HOOD  TX  76544-5073 
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NO.  OF 

COPIES  ORGANIZATION 


1  ARMY  RSCH  LABORATORY  -  HRED 

RDRL  HRM  CN  R  SPENCER 
DCSFDI  HF 

HQUSASOC  BLDG  E2929 
FORT  BRAGG  NC  28310-5000 

1  ARMY  RSCH  LABORATORY  -  HRED 
HUMAN  RSRCH  AND  ENGRNG 
DIRCTRT  MCOE  FIELD  ELEMENT 
RDRL  HRM  DW  E  REDDEN 
6450  WAY  ST 
BLDG  2839  RM310 
FORT  BENNING  GA  31905-5400 

1  ARMY  G1 
(CD  DAPE  MR  B  KNAPP 
only)  300  ARMY  PENTAGON  RM  2C489 
WASHINGTON  DC  20310-0300 

ABERDEEN  PROVING  GROUND 

4  DIR  USARL 
RDRL  HR 
L  ALLENDER 
T  LETOWSKI 
RDRL  HRM 

P  S  A V AGE-KNEPS  HIELD 
RDRL  HRS  D 
B  AMREIN 
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